{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 构建一个好的训练集---数据预处理\n",
    "\n",
    "数据的质量和包含的有用信息量是决定一个机器学习算法能够学多好的关键因素。因此，我们在训练模型前评估和预处理数据显得至关重要。在本章，我们要讨论必不可少的预处理技术，能够帮助我们构建更好的机器学习模型。\n",
    "\n",
    "本章涉及的主题：\n",
    "* 移除数据集中的缺失值\n",
    "* 将分类(category)数据转型，能够被机器学习算法处理\n",
    "* 特征选择"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 处理缺失值\n",
    "\n",
    "现实中的数据总是存在或多或少的缺失值现象。原因多种多样，可能是数据收集阶段发生错误，也可能数据调研阶段某些选项没有被填写。不论是什么原因造成的缺失值，我们都统一将其看做空格或者用NaN(Not a Number)表示的占位符。\n",
    "\n",
    "\n",
    "不幸地是，大多数计算工具不能处理缺失值，即使我们忽略缺失值也不能产生预测结果。因此，我们必须认真对待缺失值问题。在讨论处理缺失值的方法之前，我们先创建一个例子，以便更好地理解缺失值问题："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from io import StringIO\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "csv_data = '''A, B, C, D\n",
    "              1., 2., 3., 4.\n",
    "              5., 6.,, 8.\n",
    "              0., 11., 12.,'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "csv_data = unicode(csv_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(StringIO(csv_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A     B     C    D\n",
       "0  1.0   2.0   3.0  4.0\n",
       "1  5.0   6.0   NaN  8.0\n",
       "2  0.0  11.0  12.0  NaN"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "上面的代码，我们创建了一个csv格式的变量csv_data, 然后读入DataFrame对象，注意其中两个缺失值被NaN替代了。\n",
    "\n",
    "如果DataFrame对象包含的数据很多，人工来查找NaN就不现实了。我们可以使用isnull方法来返回一个值为布尔类型的DataFrame，判断每个元素是否缺失，如果元素缺失，值为True。然后使用sum方法，我们就能得到DataFrame中每一列的缺失值个数，还是看代码理解吧："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A     0\n",
       " B    0\n",
       " C    1\n",
       " D    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "现在我们知道如果统计DataFrame每一列中的缺失值个数。下一节我们学习几种处理缺失值的策略。\n",
    "\n",
    "\n",
    "**Note** 虽然scikit-learn和NumPy数组结合的很方便，但是预处理时还是推荐使用pandas的DataFrame格式而非NumPy数组。由DataFrame对象得到NumPy数组很方便，直接通过values属性即可，然后就可以用sklearn中的算法了：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  2.,  3.,  4.],\n",
       "       [ 5.,  6., nan,  8.],\n",
       "       [ 0., 11., 12., nan]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " ## 消除带有缺失值的特征或样本\n",
    "\n",
    "处理缺失值最简单的手段无疑是直接将带有缺失值的特征(列)或样本(行)从数据集中去掉。去掉行可以通过dropna方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C    D\n",
       "0  1.0  2.0  3.0  4.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "去掉列同样用dropna方法，只不过将参数axis设置为1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A     B\n",
       "0  1.0   2.0\n",
       "1  5.0   6.0\n",
       "2  0.0  11.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "dropna方法包含多个参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A     B     C    D\n",
       "0  1.0   2.0   3.0  4.0\n",
       "1  5.0   6.0   NaN  8.0\n",
       "2  0.0  11.0  12.0  NaN"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 只去掉所有值均为NaN的行\n",
    "df.dropna(how='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C    D\n",
       "0  1.0  2.0  3.0  4.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 去掉非缺失值小于4个的行\n",
    "df.dropna(thresh=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "虽然移除缺失值的操作很简单，但是这样做的缺点也很明显，比如，如果每一行都有一个缺失值，那整个数据集都被移除了。或者移除了过多的特征。\n",
    "\n",
    "下一节我们学习处理缺失值真正常用的做法：插入法(interpolation)。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 改写缺失值\n",
    "\n",
    "用移除法处理缺失值的缺点很明显，过于暴力，会丢掉很多数据信息。怎样保留那些非缺失值数据的同时处理缺失值呢？这就是插入法，用一个估计值来替代缺失值。最常用的是平均估计法，即用整个特征列的平均值代替这一列的缺失值。\n",
    "\n",
    "使用sklearn中的Imputer类能很容易实现此方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Imputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "imr = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "imr = imr.fit(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "imputed_data = imr.transform(df.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.  3.  4.]\n",
      " [ 5.  6. nan  8.]\n",
      " [ 0. 11. 12. nan]]\n",
      "-------------------------------\n",
      "[[ 1.   2.   3.   4. ]\n",
      " [ 5.   6.   7.5  8. ]\n",
      " [ 0.  11.  12.   6. ]]\n"
     ]
    }
   ],
   "source": [
    "print(df.values)\n",
    "print('-------------------------------')\n",
    "print(imputed_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "先计算每一列的平均值，然后用相应列的平均值来替换NaN。如果将参数axis=0改为axis=1，则会计算每个样本的所有特种的平均值。参数strategy的其他取值包括median和most_frequent。most_frequent对于处理分类数据类型的缺失值很有用。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 理解sklearn中estimator的API\n",
    "\n",
    "上一节我们调用了sklearn中Imputer类来处理缺失值。Imputer属于sklearn中所谓的transformer类，专门用于数据转换。此类estimator的两个必不可少的方法是fit和transform。fit方法用于从训练集中学习模型参数，transform用学习到的参数转换数据。\n",
    "\n",
    "任何要进行转换的数据的特征维度必须和fit时的数据特征维度相同。\n",
    "\n",
    "下图演示了fit和transform的过程：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576a5b856e73c.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "我们在第三章用到的各类分类器属于sklearn中的estimator，它的API和transformer非常像。Estimator还有一个predict方法，大部分也含有transform方法。同样Estimator含有fit方法来学习模型参数。只不过不同的是，在监督学习时，我们还像fit方法提供每个样本的类别信息。\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576a5cc2bd8b0.png)\n",
    "\n",
    "( 图中应该是est.fit(X_train,y_train))\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 处理分类数据\n",
    "\n",
    "\n",
    "\n",
    "目前为止，我们处理的都是数值型变量。但是真实世界的数据集通常都含有分类型变量(categorical value)的特征。当我们讨论分类型数据时，我们不区分其取值是否有序。比如T恤尺寸是有序的，因为XL>L>M。而T恤颜色是无序的。\n",
    "\n",
    "\n",
    "\n",
    "在讲解处理分类数据的技巧之前，我们先创建一个新的DataFrame对象：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame([['green', 'M', 10.1, 'class1'],\n",
    "                  ['red', 'L', 13.5, 'class2'],\n",
    "                  ['blue', 'XL', 15.3, 'class1']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df.columns = ['color', 'size', 'price', 'classlabel']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>M</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>L</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>XL</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color size  price classlabel\n",
       "0  green    M   10.1     class1\n",
       "1    red    L   13.5     class2\n",
       "2   blue   XL   15.3     class1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "\n",
    "上面创建的数据集含有无序特征(color)，有序特征(size)和数值型特征(price)。最后一列存储的是类别。在本书中类别信息都是无序的。\n",
    "\n",
    "\n",
    "### 映射有序特征\n",
    "\n",
    "\n",
    "\n",
    "为了保证学习算法能够正确解释有序特征(ordinal feature)，我们需要将分类型字符串转为整型数值。不幸地是，并没有能够直接调用的方法来自动得到正确顺序的size特征。因此，我们要自己定义映射函数。在接下来的简单的示例，假设我们知道特征取值间的不同，比如 XL=L+1=M+2。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "size_mapping = {\n",
    "    'XL': 3,\n",
    "    'L': 2,\n",
    "    'M': 1\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df['size'] = df['size'].map(size_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price classlabel\n",
       "0  green     1   10.1     class1\n",
       "1    red     2   13.5     class2\n",
       "2   blue     3   15.3     class1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "如果我们还想将整型变量转换回原来的字符串表示，我们还可以定义一个反映射字典 inv_size_mapping={v: k for k, v in size_mapping.items()}。\n",
    "\n",
    "\n",
    "### 对类别进行编码\n",
    "\n",
    "\n",
    "\n",
    "许多机器学习库要求类别是整型数值。虽然sklearn中大部分Estimator都能自动将类别转为整型，我还是建议大家手动将类别进行转换。对类别进行编码，和上一节中转化序列特征很相似。但不同的是类别是无序的，所以我们可以从0开始赋整数值:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class_mapping = {label: idx for idx, label in enumerate(np.unique(df['classlabel']))}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'class1': 0, 'class2': 1}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "接下来我们可以利用映射字典对类别进行转换："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df['classlabel'] = df['classlabel'].map(class_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price  classlabel\n",
       "0  green     1   10.1           0\n",
       "1    red     2   13.5           1\n",
       "2   blue     3   15.3           0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "得到整型类别值，也可以用映射字典转为原始的字符串值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "inv_class_mapping = {v: k for k, v in class_mapping.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df['classlabel'] = df['classlabel'].map(inv_class_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price classlabel\n",
       "0  green     1   10.1     class1\n",
       "1    red     2   13.5     class2\n",
       "2   blue     3   15.3     class1"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "上面是我们自己手动创建的映射字典，sklearn中提供了LabelEncoder类来实现类别的转换："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class_le = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "y = class_le.fit_transform(df['classlabel'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "fit_transform方法是fit和transform两个方法的合并。我们还可以调用inverse_transform方法得到原始的字符串类型值：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lj/.pyenv/versions/2.7.12/lib/python2.7/site-packages/sklearn/preprocessing/label.py:171: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['class1', 'class2', 'class1'], dtype=object)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_le.inverse_transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 对离散特征进行独热编码\n",
    "\n",
    "前面一节我们使用字典映射来转化有序特征，由于sklearn中Estimator把类型信息看做无序的，我们使用LabelEncoder来进行类别的转换。而对于无序的离散特征，我们也可以使用LabelEncoder来进行转换：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X = df[['color', 'size', 'price']].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "color_le = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X[:, 0] = color_le.fit_transform(X[:, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 10.1],\n",
       "       [2, 2, 13.5],\n",
       "       [0, 3, 15.3]], dtype=object)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "现在我们将无序离散特征转换为整型了，看起来下一步就是直接训练模型了。如果你这样想，恭喜你，你犯了一个很专业的错误。在处理分类型数据(categorical data)时，这是很常见的错误。你能发现问题所在吗？虽然“颜色”这一特征的值不含有顺序，但是由于进行了以下转换：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576b3bb04ba79.png)\n",
    "\n",
    "学习算法会认为‘green’比‘blue’大，‘red’比‘green’大。而这显然是不正确的，因为本身颜色是无序的！模型错误的使用了颜色特征信息，最后得到的结果肯定不是我们想要的。\n",
    "\n",
    "\n",
    "那么如何处理无序离散特征呢？常用的做法是**独热编码(one-hot encoding)**。独热编码会为每个离散值创建一个哑特征(dummy feature)。什么是哑特征呢？举例来说，对于‘颜色’这一特征中的‘蓝色’，我们将其编码为[蓝色=1，绿色=0，红色=0]，同理，对于‘绿色’，我们将其编码为[蓝色=0，绿色=1，红色=0]，特点就是向量只有一个1，其余均为0，故称之为one-hot。\n",
    "\n",
    "在sklearn中，可以调用OneHotEncoder来实现独热编码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ohe = OneHotEncoder(categorical_features=[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0. ,  1. ,  0. ,  1. , 10.1],\n",
       "       [ 0. ,  0. ,  1. ,  2. , 13.5],\n",
       "       [ 1. ,  0. ,  0. ,  3. , 15.3]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe.fit_transform(X).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 10.1],\n",
       "       [2, 2, 13.5],\n",
       "       [0, 3, 15.3]], dtype=object)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3x5 sparse matrix of type '<type 'numpy.float64'>'\n",
       "\twith 9 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "在初始化OneHotEncoder时，通过categorical_features参数设置要进行独热编码的列。还要注意的是OneHotEncoder的transform方法默认返回稀疏矩阵，所以我们调用toarray()方法将稀疏矩阵转为一般矩阵。我们还可以在初始化OneHotEncoder时通过参数sparse=False来设置返回一般矩阵。\n",
    "\n",
    "\n",
    "除了使用sklearn中的OneHotEncoder类得到哑特征，我推荐大家使用pandas中的get_dummies方法来创建哑特征，get_dummies默认会对DataFrame中所有字符串类型的列进行独热编码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>size</th>\n",
       "      <th>color_blue</th>\n",
       "      <th>color_green</th>\n",
       "      <th>color_red</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.5</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15.3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price  size  color_blue  color_green  color_red\n",
       "0   10.1     1           0            1          0\n",
       "1   13.5     2           0            0          1\n",
       "2   15.3     3           1            0          0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(df[['price', 'color', 'size']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 将数据集分割为训练集和测试集\n",
    "\n",
    "本节，我们使用一个新的数据集：Wine。Wine也属于UCI开源数据集，它包含178个样本，每个样本有13维度特征，描述了不同的化学属性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Wine 数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df_wine = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash', 'Alcalinity of ash', 'Magnesium', 'Total phenols', 'Flavanoids', \n",
    "                  'Nonflavanoid phenols', 'Proanthocyanins', 'Color intensity', 'Hue', '0D280/0D315 of diluted wines', 'Proline']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Class labels', array([1, 2, 3]))\n"
     ]
    }
   ],
   "source": [
    "print(\"Class labels\", np.unique(df_wine['Class label']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Class label</th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Malic acid</th>\n",
       "      <th>Ash</th>\n",
       "      <th>Alcalinity of ash</th>\n",
       "      <th>Magnesium</th>\n",
       "      <th>Total phenols</th>\n",
       "      <th>Flavanoids</th>\n",
       "      <th>Nonflavanoid phenols</th>\n",
       "      <th>Proanthocyanins</th>\n",
       "      <th>Color intensity</th>\n",
       "      <th>Hue</th>\n",
       "      <th>0D280/0D315 of diluted wines</th>\n",
       "      <th>Proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.8</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Class label  Alcohol  Malic acid   Ash  Alcalinity of ash  Magnesium  \\\n",
       "0            1    14.23        1.71  2.43               15.6        127   \n",
       "\n",
       "   Total phenols  Flavanoids  Nonflavanoid phenols  Proanthocyanins  \\\n",
       "0            2.8        3.06                  0.28             2.29   \n",
       "\n",
       "   Color intensity   Hue  0D280/0D315 of diluted wines  Proline  \n",
       "0             5.64  1.04                          3.92     1065  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_wine.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wine数据集一共有三个类别：1,2和3.表示三个葡萄品种。\n",
    "\n",
    "\n",
    "首先将数据集随机分割为训练集和测试集，一种简单的方法是使用sklearn.cross_validation中的train_test_split方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = df_wine.iloc[:, 1:].values, df_wine.iloc[:, 0].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "解释一下上面的代码： 首先，我们将特征矩阵赋值给X，将类别向量赋值给y，然后调用train_test_split方法随机分割X和y。通过设置test_size=0.3,使得训练集占Wine样本数的70%，测试集占30%。\n",
    "\n",
    "\n",
    "**Note** 在分割数据集时，如果确定训练集和测试集的大小没有通用的做法，一般我们选择60:40, 70:30或者80:20。对于大数据集，90:10甚至 99:1也是比较常见的。还要注意的是，通过本地验证得到最优模型和参数时，还要在整个数据集(训练集+验证集+测试集)上训练一次，得到最终的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 统一特征取值范围\n",
    "\n",
    "特征缩放(feature scaling)是预处理阶段的关键步骤，但常常被遗忘。虽然存在决策树和随机森林这种是少数不需要特征缩放的机器学习算法，但对于大部分机器学习算法和优化算法来说，如果特征都在同一范围内，会获得更好的结果。比如第二章提到的梯度下降法。\n",
    "\n",
    "特征缩放的重要性可以通过一个简单的示例解释。假设我们有两个特征，一个特征的取值范围是[1,10],另一个特征的取值范围是[1,100000]。我们使用Adaline中的平方误差函数，很明显，权重更新时会主要根据第二维度特征，这就使得在权重更新过程中第一个特征的话语权很小。另一个例子是如果kNN算法用欧氏距离作为距离度量，第二维度特征也占据了主要的话语权。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有两种方法能使不同的特征有相同的取值范围：归一化(normalization)和标准化(standardization)。两种方法还是有必要区分一下的。归一化指的是将特征范围缩放到[0,1]，是最小-最大缩放(min-max scaling)的特例。为了得到归一化结果，我们对每一个特征应用最小-最大缩放，计算公式如下：\n",
    "\n",
    "$$x_{norm}^{(i)}=\\frac{x^{(i)} - x_{min}}{x_{max}-x_{min}}$$\n",
    "\n",
    "\n",
    "其中，$x_{norm}^{(i)}$是$x^{(i)}$归一化后的结果，$x_{min}$是对应的列特征最小值，$x_{max}$则是最大值。\n",
    "\n",
    "\n",
    "sklearn中实现了最小-最大缩放，调用MinMaxScaler类即可：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mms = MinMaxScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_norm = mms.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_norm = mms.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然归一化方法简单，但相对来说，标准化对于大部分机器学习算法更实用。原因是大部分线性模型比如逻辑斯蒂回归和线性SVM在初始化权重参数时，要么选择0要么选择一个接近0的随机数。实用标准化，我们能将特征值缩放到以0为中心，标准差为1，换句话说，标准化后的特征形式服从正态分布，这样学习权重参数更容易。此外，标准化后的数据保持了异常值中的有用信息，使得算法对异常值不太敏感，这一点归一化就无法保证。\n",
    "\n",
    "\n",
    "标准化的计算公式如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x_{std}^{(i)}=\\frac{x^{(i)}-u_{x}}{\\sigma_{x}}$$\n",
    "\n",
    "\n",
    "此时，$u_{x}$是训练集对应特征列的平均值，$\\sigma_{x}$是对应特征列的标准差。\n",
    "\n",
    "下面一张表使用一个简单的例子，展示了标准化和归一化的区别：\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576b51a76b9ea.png)\n",
    "\n",
    "\n",
    "sklearn中提供了StandardScalar类实现列标准化：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stdsc = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_std = stdsc.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_std = stdsc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**再次强调，StandardScaler只使用训练集fit一次，这样保证训练集和测试集使用相同的标准进行的特征缩放**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 选择有意义的特征\n",
    "\n",
    "如果一个模型在训练集的表现比测试集好很多，那我们就要小心了，模型很可能过拟合了。过拟合意味着模型捕捉了训练集中的特例模式，但对未知数据的泛化能力比较差，我们也说模型此时具有高方差。\n",
    "\n",
    "模型过拟合的一个原因是对于给定的训练集数据，模型过于复杂，常用的减小泛化误差的做法包括：\n",
    "* 收集更多的训练集数据\n",
    "* 正则化，即引入模型复杂度的惩罚项\n",
    "* 选择一个简单点的模型，参数少一点的\n",
    "* 降低数据的维度\n",
    "\n",
    "\n",
    "在上面的一系列做法中，第一条收集更多数据通常不实用。在下一章，我们会学习一个有用的技巧来判断更多的训练集数据是否有帮助。在接下来的章节，我们学习正则化和特征选择的方法来降低过拟合。\n",
    "\n",
    "\n",
    "### L1正则\n",
    "\n",
    "会议第三章，我们运用过L2正则来降低模型的复杂度，当时我们定义的L2正则项：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576b57f087fe7.png)\n",
    "\n",
    "\n",
    "除了L2正则，另一个中减低模型复杂度的方法是L1正则(L1 regularization)：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576b582456836.png)\n",
    "\n",
    "L2正则项是权重参数的平方和，而L1正则项是权重参数的绝对值和。相对于L2, L1正则项趋向于得到稀疏特征向量，即很多特征权重参数为0.如果数据集的特征维度很高且特征不相干(极端情况是 不相干的特征维度数目比训练样本数还大)，特征稀疏性是非常有用的。 由于很多特征权重为0，所以，很多人也把L1正则看做特征选择的一种方式。\n",
    "\n",
    "\n",
    "为了更好地理解L1正则倾向于产生稀疏特征，让我们看一下正则化的几何解释。假设损失函数是差平方损失函数(sum of the squared errors, SSE),且只含有两个权重参数$w_{1}, w_{2}$，易知损失函数是凸函数。对于给定的损失函数，我们的目的是找到损失函数取最小值时对应的权重值，如下图损失函数等高线所示，当($w_{1},w_{2}$)取椭圆中心点时，损失函数值最小：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/23/576b8d0681fc4.png)\n",
    "\n",
    "\n",
    "而正则项是对现在损失函数的惩罚项，它鼓励权重参数取小一点的值，换句话说，正则项惩罚的是大权重参数。\n",
    "\n",
    "因此，如果增大正则系数$\\lambda$的值，也就增大了正则项的威力，也就会导致权重参数变小(趋向于0)，从而也减小了模型对训练数据的依赖。我们在下图画出L2惩罚项：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/23/576b8e1a721d3.png)\n",
    "\n",
    "\n",
    "L2正则项用图上阴影球形表示$w_{1}^{2}+w_{2}^{2}<=\\frac{1}{\\lambda}$。**正则化后的 新损失函数=原始损失函数+正则项，我们可以换一种思路解释这个公式，新的损失函数还是原始损失函数，我们把正则项看做权重参数的限制条件，也就是说，权重参数的取值范围必须在图中阴影球形内**。如果增大$\\lambda$的值，会缩小阴影球形面积。比如，如果$$\\lambda$$趋向正无穷，则阴影面积趋向0，权重参数也趋向0.\n",
    "\n",
    "我们总结一下：我们的目标是最小化原始损失函数和正则项，等价于在原始损失函数基础上增加限制条件，更加偏向于简单模型来减小方差。\n",
    "\n",
    "\n",
    "现在我们讨论L1正则项和稀疏性。L1正则和刚才讨论的L2正则很像。当然区别还是有的：L1正则是权重参数绝对值的和，我们用一个另行区域表示，如下图所示：\n",
    "\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/23/576b939fa6338.png)\n",
    "\n",
    "\n",
    "由于菱形的特点，在和损失函数等高线相交时，最小点很可能落在坐标轴上，这就导致了特征的稀疏性。如果你对这背后的数学感兴趣，推荐阅读 *The Elements of Statistical Learning, Trevor Hastie, Robert Tibshirani, and Friedman, Springer*。\n",
    "\n",
    "\n",
    "对于sklearn中那些支持L1正则的模型，我们只需要在初始化时用penalty参数设置为L1正则即可：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LogisticRegression(penalty='l1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将L1正则逻辑斯蒂回归应用到标准化后的Wine数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = LogisticRegression(penalty='l1', C=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.983870967742\n"
     ]
    }
   ],
   "source": [
    "print(\"Training accuracy: {}\".format(lr.score(X_train_std, y_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "模型在训练集和测试集上的准确率说明没有过拟合。如果我们调用lr.intercept_属性，可以发现只返回了三个值：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.38378555, -0.15813427, -0.70041216])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "由于Wine数据集是多类别数据，所以lr使用了One-vs-Rest(OvR)方法，所以上面三个值分别属于三个模型：第一个模型用类别1 vs 类别2和3；第二个模型用类别2 vs 类别1和3；第三个模型用类别3 vs 类别1和2。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.28038386,  0.        ,  0.        , -0.02798846,  0.        ,\n",
       "         0.        ,  0.71018965,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.23572666],\n",
       "       [-0.64370733, -0.06892275, -0.05715523,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        , -0.92712303,\n",
       "         0.05983122,  0.        , -0.37105833],\n",
       "       [ 0.        ,  0.06151483,  0.        ,  0.        ,  0.        ,\n",
       "         0.        , -0.63613533,  0.        ,  0.        ,  0.49813629,\n",
       "        -0.35820553, -0.5711789 ,  0.        ]])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过lr.coef_得到权重数组，共三行，每一个类对应一行。每一行有13个参数，对应13个特征。网络输入计算如下：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/23/576b9a7bc8c6c.png)\n",
    "\n",
    "我们可以发现权重向量中有很多0值，这说明L1正则可以作为特征选择的一种手段，得到的模型具有鲁棒性。\n",
    "\n",
    "\n",
    "\n",
    "最后，我们画出正则路径，即不同正则威力下的不同特征的权重参数：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAEOCAYAAAAnlgsdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPW9//HXOTOTTCaTPSFkJZCVBAghkAiyuIF4CypF\nxaWioD9X6i0I0tpe9ba1VVx6q3XvtRYKImqtvVgVEQsKyk7CGnZIQjayTDJJZjIz5/v74ySTBBLW\nkAT4Pnmcx0zOnJn5ZpTkzee7KUIIJEmSJEmSpJ6j9nQDJEmSJEmSLncykEmSJEmSJPUwGcgkSZIk\nSZJ6mAxkkiRJkiRJPUwGMkmSJEmSpB4mA5kkSZIkSVIPk4FMkiRJkiSph8lAJkmSJEmS1MOMPd0A\nSZIk6fK1efPmPkaj8c/AIGSRQLp0acAOt9t9f3Z2dnlHF8hAJkmSJPUYo9H45759+w6MiIioVlVV\nbh0jXZI0TVMqKirSS0tL/wzc2NE1l0QgCw8PFwkJCT3dDEmSpIvK5s2bjwshInq4GYNkGJMudaqq\nioiICFtpaemgzq65JAJZQkICmzZt6ulmSJIkXVQURTnS020AVBnGpMtB8//nnXbLy/56SZIk6bK3\naNGiYEVRsrdu3WoGKCgo8ElOTs44l9eKiYkZXFJScsYFj1deeSVs+vTp8efyXtKlQwYySZIk6bK3\ndOnS0GHDhtkXLlwY2tNtkS5PMpBJkiRJlzWbzaZu3LjR+pe//OXwJ598clIgc7vdPPDAA7HJyckZ\nKSkp6c8++2wfgE8//TRg4MCB6SkpKem33nprQmNjo9LynAULFvRJT08fmJKSkt5SdSsrKzNcd911\niSkpKemZmZlp69ev9+u+71Lq7WQgkyRJki5rS5YsCb7qqqtsQ4YMcYaEhLi//fZbS9vHX3rppYij\nR4/67Nq1a+fevXt33X///ZUNDQ3Kgw8+2P+DDz44sHfv3l1ut5sXXnjBO0EiPDzcvWvXrt0zZ86s\neO655yIBnnjiiejMzMyGvXv37vrNb35TfM899/Tv7u9V6r0uiUH9kiRJ0sVv5kziduzAcvorz9yg\nQTS8+y6Fp7pm2bJloY899lg5wNSpU6sWLVoU+vjjj3vXilq1alXgQw89VGEymQCIjIz0fP/9936x\nsbHOIUOGOAHuvffeytdee60PUA5w5513VgPk5OQ0/POf/wwB2LBhQ8DHH3+8H+DGG2+se+CBB4xV\nVVWyMCIBMpBJkiRJl7GysjLDDz/8EFBQUOA3a9YsPB6PoiiKmDNnToeLd54ps9ksAIxGo3C73crp\nrpckGcgkSZK6kd1up6ysDEVR8PX1xWw24+vri6+vL0ajEUW5fH93n66SdSEsWrQoZMqUKVVLlizx\nLgEyYsSI1EOHDvm0fH3ttdfWvvXWW+GTJk2qNZlMlJWVGTIzMx3FxcU+O3bs8B00aJBz4cKFYWPG\njKk71Xvl5ubW/eUvfwl74YUXSpYvXx4QEhLiDg0N1S7k9yddPGQgkyRJukAaGho4duwYx44do6Sk\nhGPHjlFbW9vp9aqqesPZiWHtxKOjx1rOGQyGbvwuL24ffvhh6Lx580rbnrvpppuqf//730e1fD17\n9uyKvXv3+qalpWUYjUZxzz33VDz55JMVb7755uFbb7010ePxkJmZ2TB37tyKU73X888/f+yuu+5K\nSElJSffz89Pee++9Qxfq+5IuPooQF/96fMOHDxdyYVhJknqSw+Hwhq6Wo6amxvt4WFgY0dHRREVF\n0bdvX1RVxeFw4HQ62x0Oh4OmpqZOH9O00xdUDAZDp4HNx8fH+9jo0aM3CyGGX8jP5XTy8vIOZ2Zm\nHu/JNkhSd8nLywvPzMxM6OgxWSGTJEk6S01NTd7w1XJbWVnpfTw4OJjo6GiGDx/uDWFms/m831cI\ngcfj6TSsnXiu7VFdXd3u60vhH+OSdCmRgUySJOkUXC4XZWVl7Spfx48f9waagIAAYmJiGDJkCNHR\n0URHR2OxdOlEQS9FUTAajVitVqxW6zm/jhACl8vFM88803WNkyTpvMhAJkmS1Mzj8bQLXyUlJZSX\nl3u7Cf39/YmOjiY9Pd1b+QoICOjhVp89RVHw8fE5/YWSJHUbGcgkSbosaZpGRUVFu8pXWVkZHo8H\nALPZTExMDKNGjfJWvgIDAy/rWZCSJF04MpBJknRZEEJw6NAhCgoKKCkpoaSkBLfbDYCPjw/R0dHk\n5uZ6w1dwcLAMX5IkdRsZyCRJuqQ1NTWRn5/Phg0bqKiowGg0EhUVRXZ2tjd8hYWFyfAlSVKPkls2\nSJJ0SaqurubLL7/kD3/4A5999hlGo5GbbrqJ+fPnM3PmTCZOnMiQIUMIDw+XYewypyhK9k033eTd\nV9LlchESEpJ59dVXJ53qecuXLw9ouWbx4sVBTz75ZN8L0b41a9ZY7r333riOHouJiRlcUlIiiyuX\nAPkfUZKkS0ZLt+T69evZu3cviqKQnp5Obm4usbGxMnhJHfLz89MKCgr87Ha7YrVaxSeffBIYGRnp\nOpvXuOuuu2yA7UK0b+zYsQ1jx45tuBCvLfUeskImSdJFr6mpiY0bN/LGG2+waNEiioqKGDNmDD/7\n2c+45ZZbiIuLk2FMOqXrrrvO9uGHHwYDvP/++6FTp06tannsm2++sQwdOjRt4MCB6VlZWWl5eXm+\nJz7/lVdeCZs+fXo8QGFhoXH8+PGJqamp6ampqelfffWV/4nX33XXXfGDBg0amJSUlDF79uzolvOr\nV6+2ZGVlpaWmpqYPHjx4YHV1tdq2EldaWmq48sork5OSkjKmTZvWT64nd+notRUyRVEOA3WAB3D3\n9GrSkiT1PlVVVWzcuJGtW7fidDqJiori5ptvJiMjA6Ox1/54k3qhu+++u+rpp5+OmjZtWs3u3bst\n9913X+W6deusAJmZmY6NGzfuMZlM/OMf/wh44oknYr/88ssDnb3WQw89FD9mzJi6p5566oDb7cZm\ns520l9XLL79cHBkZ6XG73YwaNSp1/fr1fpmZmY677rorcfHixQfGjRvXUFVVpVqt1nZbM/z85z+P\nHjlypP3FF18sWbp0adCyZcvCu/7TkHpCb/+JdbUQQm6pIUmSlxCCgwcPsmHDBvbu3YuqqqSnp5OT\nkyO7JS9yMz+dGbejfEeXrqo7qM+ghndveve0m5bn5uY2FhUV+b7zzjuh1113Xbuux6qqKsO0adP6\nHz582KwoinC5XKf8n2zdunUBH3300SEAo9FIWFiY58Rr/vrXv4a+99574W63W6moqDDl5eWZFUWh\nT58+rnHjxjUAdLTx+A8//BDw97//fT/A7bffbnvwwQdPem3p4tTbA5kkSRKgd0vm5eWxYcMGjh8/\njsViYezYsQwfPvyiXJxV6n0mTpxY8/TTT8etWLGioLy83Pv7cf78+THjxo2r++qrrw4UFBT4XHPN\nNann8z579uzx+dOf/hS5efPm3REREZ6pU6cmOBwOOYToMtebA5kAViiKIoC3hBBv93SDJEnqfrJb\n8vJxJpWsC+nhhx8+Hhwc7MnJyWlcvny5N+XX1tYaYmNjmwDeeuut03YRXnnllXUvvPBCxFNPPVXe\n0mXZtkpWXV1t8PPz00JDQz2FhYXGf//730Hjxo2rGzJkiKO8vNy0evVqy7hx4xqqq6tP6rK84oor\n6t57772wBQsWlCxbtiywtrb2pO5Q6eLUm3+ajRZCFCuK0gf4SlGUPUKINS0PKoryAPAAQHx8fE+1\nUZKkC0B2S0o9ITEx0fWrX/2q/MTz8+fPL73//vv7P//889Hjx4+vOd3rvPHGG0fvvffefikpKeGq\nqvKnP/3pyHXXXVff8vjIkSMbBw0a1JCYmDgoKiqqKTs72w5gNpvF4sWLDzz22GPxDodDNZvN2po1\na/a2fe3nnnvu2NSpUwckJSVlDB8+3B4VFdXUFd+71POUi2GGhqIozwB2IcSLHT0+fPhwsWnTpu5t\nlCRJXe7Ebkl/f3+ys7MvmW5JATQAdo8HE2BRVXwVhZ6Kl4qibO7pCVN5eXmHMzMz5Vhh6bKQl5cX\nnpmZmdDRY72yQqYoij+gCiHqmu9PAH7dw82SJOkCqaqqYsOGDWzbtg2n00l0dHSv6pYUQCNQ23zY\nOrl/0mNCYG5yEW9vIMnewEB7A0PtDfR3tBY1HIpCo0HFoao4VZUmg0qTquJSVTwGBbdqQFMVhEFF\nU1UUVQWDfquqKgaDilHVDx+Dio+q4Kuq+BpU/FQVc/M1qCrIyqIk9Vo9/5OuY5HAJ83dEkZgiRDi\ni55tkiRJXamlW3L9+vXs27cPVVXJyMggJyeHmJiYC9ItWQrs4gyC1AmP1QLu07y2IgQZjU5G2Ru4\n2t7AYHsDafYGQlytzzzu58vxAH/W9w2nSVXBoyE0DUXTUDwaqqYfRo+GUdPwc3nw8TThq2n4agKz\nR8NP0/A5x54Np6LgaA58TlWOIZek3qRXBjIhxEEgs6fbIUlS13M6nd5uycrKSvz9/Rk3bhzZ2dld\n3i1ZBawGVjUfuzq5zhcIBIKabwOBhBO+bns/WNPoW+8gwt5AkL0Bi70BY30Diqd5/LWigL8ZwoLA\natEPfwvhRgPnu2iUBtiFoNGj0aBpODUNp0e/dXk0XJqGu/nQPBqaJsDjQWhCD36ahqE5/EmS1Hv0\nykAmSdKlp7Kyko0bN3q7JWNiYpgyZQrp6eld1i1ZB3xHawDbit7daAHGAPcAw4EQWsNVIHog65Tb\nA/UNYG9z1DugpUplUMHfApHhbcKXWe8ivABUwKooWI0GQE6wk6RLhQxkkiRdME1NTezevZutW7dy\n5MiRdt2SsbGx5/36DuB7WgPYBvSuRR9gJPAMcA2Q03zu9A12tQ9e9gZodLY+bjLqgSu2TeXLz1eO\nzZIk6bzJQCZJUpcSQlBcXMzWrVvZsWMHTU1NhISEcPXVV5OVlXVe3ZIuYBOtAWwt4ESvGo0A5qEH\nsFHoVbFTNBIcTSeHr6Y2+0mbffTAFRnWGr58TDJ8SZJ0QchAJklSl7Db7eTl5bFt2zaOHz+OyWQi\nIyODoUOHEh8ff06D9DUgj9YAtgawNz+WCTyCHsDGoI/x6pTbAzY72Oqgtl7vgnS32XHGYobggNbg\nZbXo1TDpsrFo0aLg6dOnJ27ZsmVnVlaWo7PrLBZLVkNDw9bubJt0eZA/cSRJOmcej4d9+/axbds2\n9u7dixCCuLg4Jk+eTEZGBr6+pxyddRIB7KE1gP0bfWA+QCpwN3oAuwpOPTje5W4NYLY6qGvQzysK\nWP0gIrQ5ePnp478Mcsbh5W7p0qWhw4YNsy9cuDA0KyvrWE+3R7r8yEAmSdJZKy8vZ9u2beTn51Nf\nX4/VamXUqFEMHTqU8PCzm0d4iNYAtgp9aQqAeOAm9AB2NRBzqhdpcrUGsJo6qG/UzysKBPpDfJRe\nAQv0B0P3DoR3OBwcPHiQ/fv3e49Dhw7h8XgwmUwYjUZMJtNJ90/1WFfdl3Q2m03duHGjdeXKlQU3\n3nhj8h/+8IdjR44cMU2dOnWA3W43eDwe5dVXXz0yceJEO8BPf/rTmBUrVgSZzWZt+fLl++Pi4k63\nKooknZYMZJIknRGHw8GOHTvYtm0bxcXFqKpKamoqQ4cOJSkpCfUMZxUeA76hNYAdbj4fiR6+Wo7+\n0PkK9s6m5vDVHMIamnuYVFUPXQnRENQcwLphva36+noOHDjQLnS1HEVFRbTdESU4OJgBAwZgMplw\nu924XC5cLleH9088J10YS5YsCb7qqqtsQ4YMcYaEhLi//fZby8qVKwOuvfZa2/PPP1/qdrupq6tT\nARobG9WRI0faX3311eKHHnoo9tVXX41YsGBBSU9/D9LFTwYySZI6JYTg8OHDbNu2jV27duF2u4mI\niGDChAkMGTIEf3//075GLXoAW4EewPY0nw9B73p8HD2ADeQUAczh1CtfLVWwlpmPBhWCrPrA+6AA\nCLBcsABms9k6DV0lJe1/H0dERJCUlMRVV11FUlJSuyM0NPSc3l8Igcfj6TS8ncv96dOnd8VH03Vm\nzoxjx45Tzsc4a4MGNfDuqTctX7ZsWehjjz1WDjB16tSqRYsWhU6ZMqXmwQcfTHC5XOott9xSPWrU\nqEYAk8kkbr/9dhtAdnZ2/cqVKwO7tL3SZUsGMkmSTlJTU+MdoF9TU4Ovry+ZmZlkZWURHR19ygH6\nHmAL8CV6CPsefSkKf2AscB96AMukk1W0hGgfwGrq9IoYgNGgB6+oiNZB+F0467GqqqrDwLV//34q\nKiraXRsVFUVSUhITJ05sF7gSExMJCjrlFINzoigKRqOxS7eS6nWBrAeUlZUZfvjhh4CCggK/WbNm\n4fF4FEVRxJtvvlm0Zs2ago8//jho5syZ/WfNmlU2a9asSqPRKFqqwUajEbfbLafdSl1CBjJJkgBw\nu93s2bOHrVu3cvDgQQD69+/PNddcQ1pa2inHHBUBX6GHsJVAZfP5YehLUUxAX4qiw7XAhNC7HNt2\nQbYsP2Ey6gEsLlK/9fc7rwAmhKC8vLzT0FVTU9Pu+ri4OJKSkrj55pvbha4BAwZgtVrPuR1SJ05T\nyboQFi1aFDJlypSqJUuWHGk5N2LEiNTPP//cev3119sff/zx406nU9myZYuF1v+1JanLyUAmSZcx\nIQQlJSXeNcMcDgdBQUGMGzeOoUOHEhwc3OHzGoBvaa2C7Ww+3xeYhB7AxgMRHb+pPui+bRdky36P\nPia98hVk1QOYxXxOAcztdnPw4EH27NnD7t27293abDbvdaqq0q9fP5KSkrjjjjvaha7+/fvj5+d3\n1u8tXVw+/PDD0Hnz5pW2PXfTTTdVP/DAA/0tFotmNBqFxWLxLF68+FBPtVG6PCjiHDep7U2GDx8u\nNm3a1NPNkKSLRn19Pdu3b2fr1q2Ul5djNBoZOHAgQ4cOpX///id1SQpgB3oA+xI9jDnRtxwaix7A\nrgcG0cE4sJYAVl3bvAyFvXUNMF+f5gAWAMFWMJ/dqvf19fUUFBSwe/fudsFr3759uFyti7xGRUWR\nlpbGwIEDSU1NJTk5maSkJPr164ePzxmt4X9JUhRlsxBieE+2IS8v73BmZubxnmyDJHWXvLy88MzM\nzISOHpMVMkm6TGiaxoEDB9i6dSsFBQVomkZ0dDQ/+tGPGDRoEGazud31FejdkCuaj5Zh6+noC7Je\nj74g60kjsIXQB93X1EJ18zIULTME/XwhPKS1CmY+/TplLd2MHVW7jh496r3OYDCQmJhIWloakydP\nbhfAOqv0SZIk9RYykEnSJcztdlNSUkJBQQH5+fnU1dVhsVjIyclh6NChREZGeq9tAtbR2g25pfl8\nKHr34/XNtx3uQOls0itgNXX6bcsYMF8fCAuCkEA9hPl2Xo3yeDwcOnSow+BVXV3tvc5isZCWlsaY\nMWO8oSstLY2kpKTTLkTr1Jw4hANVUTG0/FEMqKioilwcVpKkniMDmSRdQmw2G0VFRd6jpKQEj8eD\noigkJydzww03kJKSgsFgQAB7aQ1g3wD16D8URgK/QQ9hw+hgNqTL1Ry+6vRKWMsyFCajHryCAyEk\noMMuyIaGBvbu3dsudLV0MzqdrRt5R0ZGkpaWxrRp09oFr9jY2E7XPGsSTdRqtdR6avXbNodNs9Ek\nmjr97BQUVFQMiuGksNZyrt3jZ/GYAUNrCGx5DANGxeg919HX3teWYVGSLnkykEnSRaql+tUSvgoL\nC6mrqwP06fjR0dHk5uYSGxtLfHw8/v7+1ACf0hrCDje/ViIwHT2AXQ2ctLCS29O6Cn51betK+AZV\nH/8VHaGHsBNmQZaXl7Nx40Y2bNjApk2b2LVrF0eOHPEulKqqKv3792fgwIHccMMN7boZO1qvyy3c\n1Gg11LpqsXls1Gl12DSbN3Q5RPstCI0YCVKDCDAEEG2MJlANxKyY0dDwCI9+i6f1vvDg/XPiuTbX\nu4QLTXT8WMt9De08/uu21xLgTgxqnQW4Mz0vSVLvIQOZJF0kamtrKSwsPKn6BRAUFES/fv2IjY0l\nNjaWvn37YjAYcAIbgGXoIWw9+obdAehrgT2BPiA/8cQ307TWNcBqavUNuUEPW0FWfSX8kEAI8PcG\nsLq6OjavXu0NYBs3buTIEX0lAVVVSU9PZ+TIkcyYMcNb7UpOTm43ds0t3NRpddRqtRQ7i0+qdjWI\nhnbNNGAgUA0kUA0k0hRJoCHQ+3WgGoif4ndOm5p3BU1onQe+E2+b77txd3jNmZxvEk36/eav3cLd\n7vquDIiSJHU9GcgkqRc6sfpVVFREbW0tcHL1KzY2loCAAEBfjuIH4G1gdfN9J/rMx+HAk+gB7Aqg\n3apiQkBdfWsFrNYOWvMM7AB/iO+rV8ACrWBQcTqd5Ofntwtfu3fv9la++vfvT25uLj/96U8ZMWIE\nw4YNw2q1ognNG7hqtVq2alupra/1Bi+7sLf7HFRUAtQAAtVA+pv660GrOXQFqUFYFEuPBa7TURUV\nFRUjxlNsQdB9hBAnBbWf8bOebpYkSc1kIJOkXuB01a/4+PiTql8AdcBa9PC1BtgIuAAVGIo+G3Is\n+mzIsLZv2HYtsJpafUHW5vfD309fCT8kEIIC0FSFgoICNvzza28Ay8vLo6lJH48VERFBTk4Ot912\nG8NGDSNlaArGICN1Wh12zU6dVsfn4nPqbPrXgtaldhQUrKqVQDWQOFOcN2i1BC9/xV+On+oiiqJg\nxIhRkT/2O7Jo0aLg6dOnJ27ZsmVnVlaWA6CgoMBn0qRJyfv27dt5uuefaOrUqQmTJk2yzZgxo3ra\ntGn9nnjiibLs7GxHZ9cvWLAgwmKxaLNmzap85ZVXwm688cbahIQEV2fXn6kvvvjCOmvWrH5Go1Fs\n2rRpt9VqPae1rnJyclJffPHFwrFjxzac/mrpXMi/mZLUzdxuN6Wlpe0CWNvqV1RUFDk5OcTFxbWr\nfgFUA5+hh6/V6DMhNfS/yMOBOegB7Eqg3eY9bbcjapkN6WqzFEWfEAgORARZKSwrbVf52rRpkz47\nM8RCdHI0w8cN50dzfkRsaizB0cEIP4Fd2LFrdg42/6G5h9OAAatqxapaiTHGEKgGEqAGeEOXVbVi\nUORYJqnnLV26NHTYsGH2hQsXhmZlZR3rytf+4IMPjpzumieeeMK7N9ff/va38KFDhzZ2RSBbuHBh\n6Jw5c0oeeeSRqvN9LenCkoFMki6wc61+AZSjD75f3XxsR1+k1RfIRe+CHIc+K7LdNt9CgL1R73qs\ntevjwVr2g/Qx6dWvkECqhYcf8rawcdVGdh3ZxdHjR8EfgmOCCUsII+f6HG6IuwGfYJ+Tflo00ICi\nKgQoAUSqkSSZkghQA7wBLEAN6NExXL2Ro8HF/k3H2L+5BEe9C6EJNE0gmg+tk9uuOcdJ5ySdzWZT\nN27caF25cmXBjTfemPyHP/zhpEDmdrt55JFHYr/55psgRVHEPffcc/yXv/xl+dy5c6O++OKLYKfT\nqQ4fPty+ePHiIyfOAm5bXbJYLFn33Xdf+YoVK4LMZrO2fPny/XFxce45c+ZEW61WT//+/Zt27Nhh\nmT59+gCz2aw988wzxX/+85/DV65ceQDgk08+CXz99dcjvvrqqwNt3+PTTz8N+PnPfx7n8XjIzMxs\nWLhw4ZE33ngj7LPPPgtdvXp10BdffBH0z3/+s91uA9ddd11iSUmJj9PpVB966KGyuXPnHne73Uyb\nNi0hPz/fX1EUcddddx1/+umnywHef//9kEcffbRfXV2d4c033zw8ceLE9mMMpPMiA5kkdREhBLW1\ntZSUlFBSUkJpaSklJSXemY8Gg4Ho6OhOq18Ax2gNX2uA3c3nLeh7Qf43egDLAdot49rk0gfe19r1\n27p60DQEAqdZoSHESJUFCmqK2FtxiOOVx3EYHPiE+BCcHkxQbhAjm//o3wz4q/7eKtaJQStADejV\n47d6AyEEpQer2fN9EXt+KGbP90UcyitD85wchBRF71JUFaXdbbtzagfnFAVFPc116gmv5328Bz6U\nXmrJkiXBV111lW3IkCHOkJAQ97fffmsZM2ZMu665l156KeLo0aM+u3bt2mkymSgrKzMAzJs3r/zF\nF18sAbj55pv7L126NOjOO++0dfQ+AI2NjerIkSPtr776avFDDz0U++qrr0YsWLCgZd1lZsyYUf3G\nG2/0aQlwmqbxi1/8IvbYsWPG6Oho97vvvhs2Y8aMdjsbNDQ0KA8++GD/FStWFAwZMsQ5ZcqUhBde\neCHiqaeeKl+7dq21pev0xLYsXrz4cGRkpMdutytZWVnpP/nJT6r37dvnW1JSYmrppj1+/Lj3X4hu\nt1vZvn377g8++CDo17/+dfTEiRP3nutnLp1MBjJJOgeaplFVVdUufJWWltLYqC8HoSgKYWFhJCQk\nEB0dTVxc3EnVL9CXnWgJX6uBln/yBgCjgXvQA9gw9I25NaHRqDVir6+ior6GRoeNRmcdjaKRRqOL\nRqObxj4e7FFN1OPEZXTrA8paWIFosBy3YLab8Vf8iXRE0j+wP+GWcG/wkmO3zl6jvYl9G4+xZ10R\ne1YVUbC5GJtN/53up/qQTDS3aleSRiwpRGPFDwUFBVCEAj1QsHqRmd3/pqcwE+J2dLD5w/kYBA3v\nwik3LV+2bFnoY489Vg4wderUqkWLFoWeGMhWrVoV+NBDD1WYTPp0mMjISA/A559/HvDyyy/3dTgc\nak1NjTE9Pb0R6DSQmUwmcfvtt9sAsrOz61euXHnSKjNtqarKbbfdVvnOO++EPvroo5Vbtmyx/v3v\nf29X6crLyzPHxsY6hwwZ4gS49957K1977bU+6EX2Tj3//PORn332WTBAaWmpaefOneYhQ4Y4CgsL\nfe+55564yZMn26ZMmVLbcv2tt95aDTBq1Kj6efPmXb57jl0gMpBJ0mm43W4qKiraha+ysjLvXokG\ng4E+ffqQlpZGVFQUUVFRREZG0vKDu0XLQqwt4WsNUCzcWEUjMVojY0Qjj2qNJIpGwkQjDq2RRtFI\noaeBvZ56GoUDh9LUOmPPTLsymWjUsFfaqdxbRW15LfZKO/bjdoxuI2H+YST0SWDwgMHkDsklOjX6\ngn9ulzI62uQeAAAgAElEQVQhBMd2VbHn/4rY800xBflFHC4rR2ueZRpLGCNIJo1YUiNjiM+MwJCh\n6vtOpQOp6H3MLSFMnHC/O84JIO4cP4BLSFlZmeGHH34IKCgo8Js1axYej0dRFEVomlZ0uuc2NDQo\njz/+eL/169fvSkpKcs2ZMyfa4XCc8l8yRqNRtHRpGo1G3G73aWuVDz/8cOWPfvSjJLPZLCZPnlx9\n4s+Wc7F8+fKA1atXB2zatGlPQECAlpOTk9rY2KhGRER4duzYseuTTz4JfPPNNyM++OCD0A8//PAw\ngNlsFi3t9ng8ssbaxWQgk6Q2nE4nZWVl7bocKyoq0DR9DScfHx/69u1LVlaWN3yFh4d7K19CCBzC\nQbWwccxVT4lWz37RSKHWSKVoRBWNWJtD11CtESMnrxx/CDgsFPw0H/zcRvxcBsLdJsxuC1qjRlVF\nLXv3HGLNt+vZnr8H+3E7DVUNRIRHkJGRQUZGBrmDcskYod+X+zieJyc0bHGy97Nj7FlbRMGeYgoq\niqj16NVQC76kEsO04NGkJseSmhtDQLafHrzS6GCVXakzp6tkXQiLFi0KmTJlStWSJUu8A+9HjBiR\n+uWXX1oHDBjg/Qt67bXX1r711lvhkyZNqm3psmz5e9+3b1+3zWZT/+///i9k8uTJJ3UNni2r1eqx\n2WzecnpCQoIrMjLS9dJLL0V98cUXJ3UTZmZmOoqLi3127NjhO2jQIOfChQvDxowZU3eq96ipqTEE\nBQV5AgICtK1bt5rz8vL8AUpKSoy+vr7avffeW5ORkeG4++67B5zv9yOdmV4byBRFmQj8EX3Xlj8L\nIZ7r4SZJl5j6+npv6Gq5rapqnYhksViIiooiKSmJvn37Etw3BGeQiWM0UKHVs0mrxy524WisR9Pq\nUUQ9vloDBjwnvVckBiIUP3xVP0IVP8INQfgpfnrocqr4NYCfXcPP5sbiNuLrMWJrbCTv4H6+3vQD\nqzatZ3PBHhxNTsLCwrzB6+ZZt3rvh4eHd+fHd+lpAApA2yEo/q6SPRuLKDhYzB5bEUco9xaX4n3C\nyY1JJS0jlrSxMcReG44hXT1hVoV0sfjwww9D582bV9r23E033VT9t7/9LfSpp57ynp89e3bF3r17\nfdPS0jKMRqO45557Kp588smKu+66q2LgwIEZERER7szMzPquaNP06dOP//SnP+03b948rWWpittv\nv73ytddeMw4bNuykpTMsFot48803D996662JLYP6586dW9HRa7eYOnWq7e23344YMGBAxoABAxwt\nbT98+LDpvvvuS9A0TQH49a9/fdpKodQ1lJaFHHsTRVEM6L0744Ei9OWV7hBC7Oro+uHDh4tNmzZ1\nYwuli4kQApvNdlL4ahlsD+AXFIA5MgQirbgjzLgijDT5u0A0YNDqMYsGDB2sdF6v+FKn+ONU/fGo\n/qiKP76qP1bVn2DFnwjVn2GqhURMqC1rf9XW03S8Ck9NHX7N/Y9uj4fth/azZttWvt+Vz/c7t2Nz\nNJKRkcGgQYO8oSsjI4PIyEg5mP582NFnS+zSj/o8BwV5xRSUFrOHIgooxo7+O8/fZCY1Poa0oTGk\nXRdLys0xWPuaT/XqFxVFUTYLIYb3ZBvy8vIOZ2ZmHj/9lZe36dOnx2dlZTXMnj1bflYXsby8vPDM\nzMyEjh7rrRWyHGC/EOIggKIoS4Gb0H+E9korPlrW003oVUy+vqgmH4Rq0LdsVo2gqnqVQQjEaQ7N\ne18PVIjm6fot5+ns+pbroMHlpKqmGntNLR5nc9VKASXQhBJuRO3njykCjOEqqlmlCRstY3E9diPC\n5ovQfFA0f1xaKAbNF4vHRKDHl2DNh1CPLyEYsNAyrKv5HzcCoBFoRHjKqa+q5qCjiWBfC75G/a9c\naWUlG/fsYmPBLnYcOYhThcT+/UlLTuaOn9zHr1OT6dtR8GrQsB8q4aIkgCb0lWubzuFwoW87cOLt\n2VzrguoqD3uajrOHIvYoxRSKCgT6TMf42AiuzB1I6vhYBo6JISY1HFWV4VfqWRkZGQP9/Py0t956\nq9u7dKXu01sDWQztxxIUoS+71KHNmzfLioF00ft283c93YTLjzjhfmHz8VHPNEeSOrJz587dp79K\nutj11kB2WoqiPAA80NPtkCRJkiRJOl+9NZAV035CdmzzOS8hxNvoeyijKErvGwgnSZIkSZJ0hnpr\nINsIJCuK0h89iN0O3NnZxdnZ2chB/ReXJo+GzemixuHC5nRTWe+i3u32rrHlccHxYhOHdpvYscnI\nwV1GKg878DdUExFYSUpCJakJlQyIqSQ2opK+IccJtVYS6FuJL5UoTZXQUAjCAyFDIeFuSLgD/KL0\nbYWOHYPt22H7dsT27Wj521B270Ft0tcW0wwqZdFBHIy1sD1SYWOogzVB1RwI8CDarDKkoBBpjSQm\nIIbogOjW28DWr5NCk/Az+fXApyydHyewHyjo4Gi7soEvkIS+uNiJR0g3tvfsyaEektR79MpZlgCK\novwH8D/oy168K4R4trNr5SzLS4MmBLVONzanHtJaAptLax0s73EYqCs3UXzQyN58E1vWmtiVr9K6\nWir4+kK/fjA4uZwb0j/g6v6LGBC0EU2oHG0azxH1J9RYp2AN9icoCIKCIDgYgiwufI7s8wY173Go\ndVFszepPQ+oAKhOjKewXzN5oX/IiNPZRybG6YxTXFXO8of0kKFVRSQ5NZkjkkHZHv6B+8hfiRUkA\nx+k4qB0A3G2ujaDjoDYAOP/FPc+XnGUpSd3rVLMse20gOxsykF26hBA0uttX02xOF/Wu1rW+TKqC\nj2bCaTNRWWzk8B4Tu7YaKSpUsNnAZoNIvz38eNjf+MmVfyMh4gh2hz9/3/hjFn13N6t2XoMm9DUY\nzWa8Ia3liLTUMVDbSbJjOwm1+URXbie8dDvm+tY1y9yR0WjpgzEMHYw2ZCAViX05EmWh0FnOropd\n5Jflk1+Wz4Hq1v2AA30DGdxncLuQNrjPYAJ82+9vKV1MXOhL+xYAe2gf1touC6UCfuidFIbm23M5\nzue5RhRlvgxkgKIo2TfeeGPVp59+egjA5XLRp0+fzKFDh9Z/8803+3uqXYcPHzY99NBDcV988cXB\nnmqD1LVkIJMuOS6PRq3TTY3T1a6i1lJMUxUI8jXR19+XKKuZIF8jHo9CrU3DWfQdPsWLCLJ9iFHY\naBDR7HHcyebKu9l3fIg3xHV02O0tLRBEUcJgtnuPIeQzkN2YcQLgxkChOYVdcRMouvJ2/K/JJTbJ\njid0Jwfs+d6Qll+Wj83ZuvVd/+D+J1XTEkMSMajt98GULjbVtIaz/ehLo7g7ODydnD/dcSbPc7Vr\nkaIgAxlgsViy4uPjnS2LsC5btizwqaeeio2KimrqyUAmXXrOK5ApitJfCHHodOd6kgxkEuhdnvYm\ntzecHW9oosqh/wKyGA1EWfVwFm7xQVUU8Dig+P/g0CI49jkINwQPgYSfQMKdYIk56T3cbqit7Tyw\n1Va58Tmyj8Aj2wkv2U5M+RaGVn6Nr3BymH4s4zaWcjvH+mSRmqaQmgopKYLQAYW4Q/MpV/LZUaGH\ntILKAjShL0brZ/RjUJ9BJwW1UL/Qbv2MpUuBRktAUxR/GcjQA9nMmTPLs7OzG2bMmFE9ZcqUhPT0\ndMe6deus33zzzf5vvvnGMnv27Hin06mazWbtvffeO5SZmemsq6tTp02bllBQUOA3YMAAR1lZmelP\nf/rT0bFjxzZYLJas++67r3zFihVBZrNZW758+f64uDj3sWPHjDNmzOhXXFzsA/Dyyy8fnTBhQv1n\nn31mffzxx+NBH9u3bt26PeXl5cZJkyYl79u3b+crr7wStmnTJv+FCxceBbj66quTHn/88bJJkybV\nWSyWrLvvvrvi66+/DurTp4/r2WefLZo/f37csWPHfJ5//vmjd911V6ebnUvd63wXhv0YGHbCuY+A\n7PNslyR1KVVRCPQ1EehrIg59EL3D7aHU7qSk3sFhWwMHahowqQqRzZWzyJip+MTfCo7jcPQDPZxt\newK2zYe+1+qTAeJ+DCYrAEYjhIbqR8eMwMDm4zb9VG0t7o8/Jfy9pcxd+wee8LxAmSOJzw/ezv/m\nT+OdmkFAPBCPyTSJxERITYXrUxoJ6L8bd1g+VT757K/N59OCT/nfrf/rfbeYgJiTQlpqWComQ8+P\nT5J6KxXwaT6kFnfffXfV008/HTVt2rSa3bt3W+67777KdevWWUHfK3Ljxo17TCYT//jHPwKeeOKJ\n2C+//PLACy+8EBEcHOw5cODAzo0bN5pHjhyZ0fJ6jY2N6siRI+2vvvpq8UMPPRT76quvRixYsKDk\nwQcfjJszZ07Z9ddfb9+3b5/P9ddfn3zw4MGdL730Ut9XXnnlyIQJE+ptNptqsVi08vLyM2p7Y2Oj\n2rzXZtH48eMTf/WrX8V8++23e7ds2WKeMWNGfxnILg6dBjJFUdKADCBIUZQft3koELh09g6RLmlm\no4GEYAsJwRbcmqC8wUmJ3UGp3UlRnQMFiLD4EGX1I6r/g1hSHoXavXD4b3Dob/DDPbDxYYibolfO\n+l4H6llOTg4MxDjjbqwz7obKSvjkEyI/+IB7V/2Oe7Xf4k7L4NiYaWwaMI1NtSkUFEBBAXz+uR9N\nTcNo+fdQSAikpAquSS/Df0A+7vB8anzyOVKbz8qDK3FpejXQpJpIj0hvF9KG9h1KH/8+XfvhSlJX\nm0kcO7B06WsOooF3T79peW5ubmNRUZHvO++8E3rddde1CzBVVVWGadOm9T98+LBZURThcrkUgHXr\n1ln/8z//sxxgxIgRjpSUlIaW55hMJnH77bfbALKzs+tXrlwZCLB27drAffv2eadd2+12g81mU6+4\n4gr73Llz42677baqO+64ozoxMfHkvdo6YTKZxC233FILkJGR0ejr66v5+vqKnJycxpZKnNT7neo3\nSyowCQgGJrc5Xwf8vwvZKEm6EIyqQrTVTLTVjBCCKoeLEruDEruTvPJa8sohyNdIlDWKqJT/InjQ\nMyjH18HhRXBkGRxeDOa+0O8O6H+3vpzG2c6SDAuD++/Xj7Iy+PhjjEuXEv/OU8TzFD/OyoLbb4eX\nbsMTl8CRI3gDmn4ofPdFX44d6wtMAEBVIT7BRUxmAdbEfESffGqa8vlq/yoW5S/yvnViSCJXxl/J\nqNhRXBl/JekR6aiK2klDJenyM3HixJqnn346bsWKFQXl5eXe34/z58+PGTduXN1XX311oKCgwOea\na65JPd1rGY1Goapqy33cbrcC+kSlLVu27LZYLO3GC/3ud78rvfnmm22ffvpp0JgxY9I+++yzfRaL\nRWv7eprWmtGcTqfa9rGW91JVFV9fXwFgMBjweDxyKvdFotNAJoT4FPhUUZSRQojvu7FNknTBKYpC\nmJ8PYX4+DIoAe5ObEruDY3Yneyrt7Km0YzaqRFkHE5XxRyKy/oih9F96l+a+P0HBHyAoQw9mCXeB\nJfbsGxEZCY88oh9FRfDhh/DBBzB/Psyfj+GKKxgwbRoDbr2VG25oP56trg727tUPPaiZKCgYxLYV\ng6ivb12yzxJWSXTWdoIGbkLxX8e/9n7BwryFAAT5BjEybqQ3oOXE5GD1sZ7X5ypJ5+UMKlkX0sMP\nP3w8ODjYk5OT07h8+XLvdOfa2lpDbGxsE8Bbb70V3nJ+5MiR9qVLl4ZMnjy5bvPmzea9e/eedsHB\n0aNH1/7+97/v85vf/KYMYN26dX6jRo1q3Llzp29OTk5jTk5O4+bNmy07duww5+TkeCtuiYmJTe+8\n847F4/Fw6NAhU35+vn/XfvdSTzuTvpf9iqI8CSS0vV4IMfNCNUqSupvVx0hyqJXkUCtOt0ZpvV45\nO2pr5FBNA0ZFIdL/aqIyJ9J3eD0+xR81jzf7OWz7BURerXdpxk8FU+DZNyA2FmbP1o9Dh2DZMli6\nVP96zhwYM0avnE2dCn36EBAA2dn60ZYQUFzctqIWRkHBVWxachWVlQCCQWMO0n/cWpTodRywreXp\n/U8jEBgUA5l9M70BbVTcKOKD4rvi45Wki0JiYqLrV7/61UkDt+bPn196//3393/++eejx48fX9Ny\nft68eRW33XZbQmJiYkZiYqIjKSnJERIS4jnx+W29/fbbhffff398SkpKusfjUXJzc+tGjRp1dMGC\nBX3WrVsXqCiKSE1NbbzllltsR48e9Q4GHT9+vP21115zJiUlZSQlJTnS09MbTvU+0sXnTGZZrgO+\nBTajz6sGQAjx8YVt2pmTsyylC8WjCSoanJTY9bFnDo+GAoT5+RBl9SWaIvyLlupjzuwHwOAHsTfp\nkwGiJpz9eLMTFRToVbOlS2H3br1/8tprYdo0mDLlVLML2tE02LIFvvgCvvwSvv8ePB4ICIAx42sY\nMPYHiFvLLvs61hetp95VD0BsYCyj4kZxZZwe0DIjM+WEgUuIXBj2/LjdbpqamhSLxSJ27tzpO2HC\nhJQDBw7sMJvNF/96UtIFcb7LXmwTQgy9EA3rKjKQSd1BCEG1w0VJvZOSOge1TfqK7AE+RqL8fYhv\nyiPg2PsoR5dBUxWY+0D87TBwLvjHnebVT/vmsGNHazg7cABMJpgwQa+c3XgjBJ55Zc5mg1WrWgPa\nkSP6+eRkGH+9m+Qx+Xii1rGpfC3rCtdx1HYUAIvJQk5MjjegjYwdSYhf794eSOqcDGTnp7q6Wh0z\nZkyqy+VShBD89re/Lbrttttqe7pdUu91voHst8A6IcS/LkDbuoQMZFJPqG9y6+HM7uB4QxMC8DWo\nRFsU+tX/m5CSD1COLQejFUb+FWImdc0bC6GXu5Yu1QNaYaG+X9R//Icezn70I/A/8+ElQuhj0b78\nUj+++QYaG8HHB0aPhuuvh6Fji6i2rmNd4VrWFa1ja8lWPEIvmKdHpHsD2pVxV5IUmiS3hLpIyEAm\nSd3rfANZHeAPNDUfCiCEEOcwUObCkIFM6mlNHo2y5nBWWu/ErQkMCsQrRQzZez8GWz4MnAeZz4La\nhV1+mgY//KAHs2XLoLQULBa9YjZtGkycqO8HdRYcDvjuu9aAtn27fr5vXz2cXX89jLqqnkPOjaw9\nqge0dYXrqHHoQ2siLBGMihvlDWjZ0dmYjXKlnN5IBjJJ6l5y6yRJ6kaaEFQ0NFFid1Bc5wBPI9dW\n/Q7zoXcgfCRcuRT8L8BgeY8Hvv1Wr5x99JG+5llgINx8M/zkJ3DddWe/TAf6JIEVK/Rw9tVXUFWl\nv0x2th7OJk6EETka+2t2s65wHWsL9W7OfVX7AH1dtOzobK6Mu5IrYq8gNyaX2MBYWUXrBWQgk6Tu\ndb4VMgW4C+gvhPiNoihxQJQQYkOXt/QcyUAm9Vb1LjffFlbR5NG4mpUEbH0YVJ+u7cLsiMulDxL7\n4AP4+9/1QWODBsHcuXDHHXp/5DnweGDzZj2cffGFXpzTND33XXttawUtIQHK68v5vvB7b0jbdGwT\nTo++z2eUNYrc2FxyonPIjc1lRPQIual6D5CBTJK61/kGsjfQNz+7RggxUFGUEGCFEGJEl7f0HMlA\nJvVmDS4P3xZW4nRrjA6uJHTTXVCTpw/2z/xd13ZhdsTp1IPZCy/oEwNiYuA//xMeeACCgs7rpWtq\n4OuvWwNaYfMqUqmpreFs3Dh9SJvT7SSvLI8NxRtYX7ye9UXrvVU0BYX0iHRyY3LJjc0lNyaXjD4Z\nGM93lqp0SjKQSVL3Ot9AtkUIMUxRlK1CiKzmc3lCiMyub+q5kYFM6u0a3Xooa3RpjIryI2LPL2Df\nGxe2C/NEQujJ6YUX9OpZQAA8+KAezmLPYWHbDl5+z57WsWf//rc+Hs3HR19GbeJEPaANGtTac1rV\nWKUHtKL1rC9ez4biDVQ2VgL6jM7h0cO9VTTZ1dn1ZCCD0tJSw1VXXZUKcPz4cZOqqiI0NNQNsG3b\ntt0nLmFRVlZm+Otf/xr6xBNPVJzqdV0uF6GhoUPr6uq2nWlbIiMjh+zcuXNneHj4KdcyOx/d8R5S\n5843kK0HRgEbm4NZBHqFLKvLW3qOZCCTLgYOt4fvCquwu9yMjAkl8vg/YP3/09cqG7nwwnZhnmjL\nFnjxRX0igKLo3Zhz58KQIV32Fg6HPqStZWmNnTv189HR+mod118P48fru0m1EEJwsPqgt4K2vng9\nW0u30uRpAlq7OnNj9GN49HDZ1XkeZCBrb86cOdFWq9Xz61//uqyza3bs2OF7yy23JO7Zs2fXqV5L\nBjKpI6cKZGeykd0rwCdAH0VRngW+A37Xdc2TpMuD2WhgTFwoAT5Gvi+uojTsJpi4Gfz7werJsHUe\nNG8QfsENGwZLlsD+/fDoo/o4s8xMvYz19dd6ues8mc164HrpJb2ntLAQ/vd/9aU0Pv1Uz4AREZCb\nC089BWvXgsejkBiayJ2D7+SPN/yRH+7/gdqf17L+/vW8MvEVrh1wLTvLd/KLr3/BNQuvIei5IAa9\nPoj7Pr2Ptze/TV5pHm7N3QUfkCTBr371q8jk5OSM5OTkjGeffbYPwNy5c2MOHz5sTktLS3/kkUdi\nqqqq1CuuuCIlPT19YEpKSvr7779/ynEAO3bs8E1KSsqYNGnSgAEDBmT8x3/8xwC73e4t+/7ud7+L\nHDhwYHpKSkp6fn6+L4DNZlOnTp2aMHjw4IEDBw5MX7JkSRDAyy+/HD5x4sQBo0ePTu7Xr9+gRx99\n1LvH2uuvvx6akpKSnpycnDFr1qyYE9tRXV2tjh07Njk1NTU9OTk54y9/+YtcULCHnXaAhhBisaIo\nm4Fr0Ze8uFkIsfuCt0ySLkG+RgOj48JYW1jJD8eqyYmOI3rC97BlDux+Ecq/g9EfdE8XJuij7//n\nf/RE9Oab8Mor+mzMrCyYNw9uvRWMXTOOKzYWZs7UD48HNm5s7d589ln4zW/0IW1tJwf06we+Rl9y\nYnLIicnxvtaJXZ2fFnzKu9veBVq7OluqaDkxObKrUzprq1at8v/www/Dtm7dusvlcinZ2dkDx48f\nX/fiiy8W33LLLeaWCpnT6VT+9a9/7Q8NDdWKi4uNo0aNSrvjjjtsp3rtAwcOmN96663D1157bf2U\nKVMSXn755YinnnqqHCAyMtK1e/fuXb/97W/7PPfcc5FLliw5On/+/Ojrr7/e9vHHHx+uqKgwjBgx\nYuDNN99cC7B7927L1q1bd/n4+IikpKTB8+bNK/d4PDz77LMxmzZt2h0aGuoZPXp0yvvvvx/Utl0f\nffRRUFxcnHPNmjX7ACorKw0X7tOUzkSnP2kVRQkUQtQqihIKlAPvt3ksVAhR1R0NlKRLja9B1UNZ\nURXri6vJiQ4mZsTr0Gec3oX5+VC44q8QO7n7GhUaCk8+qe+b+be/6d2Zd94Jv/iFvp/mffeBtes2\nHjcY4Ior9OPpp6G6Wi/MtXRv/v3v+nVpae0nB1gszc31C2Vi0kQmJk0EOu7q/OP6P3bY1Tm071Di\ng+KJC4yT3Z29zsw42GHp2tcc1ADvnvWm5atXr7ZOnjy52mq1CkDccMMNNatWrbJOmjSp3Ur8Qgge\ne+yx2A0bNlhVVaW0tNSnpKTEGB4e3mmpNiYmpunaa6+tB7j77rur3n777XD037Pceeed1QA5OTn1\nX375ZRDAv//978BVq1YFvvzyy1Ggh8D9+/f7gL5ZeWhoqAYwYMCAxgMHDvgUFhaaRo0aVRcVFeUG\nuO222ypXr14d0DaQZWdnNz7zzDOxjzzySMzNN99cM2HChPqz/YykrnWqf/ouASah72HZtv9Caf56\nwAVslyRd0nwMKqNjQ1lbVMWGYzWMiILYftMgZBisvQ3W3Nh9szDbMpvh/vv1Mtby5foEgJ/9DJ55\nBh5+GB57TF8htouFhMAtt+iHEPq2nS3Vs7fegj/+Ud+MYOzY1oCWkdE6OUBR9K7Olu5OaJ3Vub5o\nPRuO6dW0f+z5R7v3DfINIi4ojrjA5iMojtjAWO/9uMA4/Ex+Xf79SpeO119/Pay2ttawc+fOXSaT\nicjIyCENDQ2nLMcqiiJO+Np738/PTwAYDAY8Ho8Ceuj75JNPDmRkZDjbPm/lypUBvr6+3tcyGAy4\n3e4zKgUPGzbMsXnz5l0ff/xx0C9+8YvYVatW2Z577rnSM3mudGF0GsiEEJOab/t3X3Mk6fJhMqhc\nGRfK90XVbCipQROC+KBkOKkLc6k+zqw7qaq+2v+NN+qLjb34Ijz3nD4g7O679QkAaWkX5K0VBdLT\n9WP2bH0bpzVrWgPa3Ln6ERPTfnLAifusd9bVuatiF4W2QgprC1tvawvZdGwTFQ0nT5wL8ws7KbTF\nBTYHt6A4YgJi8DX6XpDP4vJz9pWsC+Wqq66qe+SRRxKeeeaZUo/Ho3zxxRfB77///sGgoCBPfX29\nd/y1zWYzREREuE0mE5988klgeXn5af8FVVxc7Lt69WrLuHHjGhYvXhw6atQo+6muv/rqq2tfeuml\nPu++q38+a9eu9bvyyisbO7t+zJgx9b/85S/jSktLDWFhYZ6PPvoodPbs2e0mKhw6dMjUt29f96OP\nPloVGBioLV68OLSz15O6x2kHhyiKMgVYJYSwNX8dDFwlhPjHqZ8pSdLpmFSVUbEhfF9czaZSGxqQ\nEGSBEa9Dn6tg/f3weVb3d2G2dcUV+sr/+/fDyy/DX/6ij86fPFkfZzZ69DntAHCm/Pxaq2KgTw5o\n2Tngk0/05qgqjBjRel1OTsdD30L9QhkdP7rT93K4HRTXFrcPa823R2xH+O7od1Q7qk96XqR/5ClD\nW3RAdK9YU00IgSY03JpbTn44jauvvrph6tSplVlZWekAM2fOrMjJyWkEGDx4cENKSkr6ddddZ/vl\nL39ZdsMNNySlpKSkDx06tL5fv37OU78yDBgwwPHiiy9Gzpgxw5KWltY4e/bsUy6hsWDBgmMPPPBA\nXEpKSrqmaUq/fv0cX3/99YHOrk9MTHT98pe/LB47dmyqEEKZMGFCze23395uXNuGDRss//Vf/xWj\nqkYqIVUAACAASURBVComk0m88cYbR87sk5EulDNZ9mKbEGLoCee2ymUvJKnreDTB98VVlDc0kRUZ\nRP/g5mE0dfvhu1uhehukPQ5Df9+9XZgdqaiA116DP/1J354pN1cvWU2Zog8O60Zud/vJARs26DsH\nBAfrkwNa1j6Li+u696xvqqewtpCi2qIOK22FtkLqmuraPUdVVKKsUd6gFhMQg1E1eoOR9xDuk891\n8dHOM8hlL7rZmS6bIV2azncdsnwhxJATzm0XQgzuuiaeHxnIpEuBRxOsP1ZNab2TzD6BJIb4Nz/g\ngC2Pw77XIeyKnunC7EhDA7z3nl41O3AAEhP1SQH33ts6+r6bVVW17hzw5ZdQVKSfHziw/eQAvws8\nLMzmsJ1UZSuqaw1wxbXFCARG1dj1h3Jm1xlUA/NHz5eBrJvJQHZ5O99A9i5QA7zWfOpRIFQIcW8X\ntrHt+z0D/D+gpYT7pBDiX6d6jgxk0qVCE3ooK7E7GRwRQHJom5mNR5bpXZiqsWe7ME/k8cA//qFP\nAFi/HsLD9bXNHn1UX2ish7RMDmiZublmTevOATk5+gSBsWNh1Ch904LLkVwYVpK61/kuDPtToAn4\noPlwooeyC+kPQoihzccpw5gkXUpURSE3OoSYADPbK+ooqGwz1rffbXDDFvBP0GdhbpnbfQvJnorB\nAFOnwvff66ln5Ej47/+G+Hh45BF97FkPaJkcMGeOHsiqqvRw9rOf6V2dCxboXZrBwfr4s8cf1xes\nrazskeZKknSZO22FrLs1V8jsQogXz/Q5skImXWo0IdhUUkNRnYOBYVYGhrcp4fTWLsy2du/WZ2Qu\nWgQuF/z4x/o4syuu6OmWedXXt2bINWv0yaTO5uHYgwe3VtDGjIGoqJ5t64UiK2SS1L3OqctSUZT/\nEUL8TFGU/6P9OmQACCFu7NJWtr7vM8C9QC2wCXhcCHHytKY2ZCCTLkVCCDaX2jha20hqmJX0MGv7\n1eZ7axdmW6Wl8Oqr8PrrUFOjp5t58+BHP9KnRvYiTqc+QaAloK1dC/bmAmVycmtAGztW3+DgUiAD\nmfT/2bvzuCjLtQ/gv3v2AYZ93xEYYBhEBAFNxRRNS+m4a6755nrM0lzaXMqOmW92TlQnzcyy18os\nzd0yF1DJBVQUhj3BhX3fl2Hu94+HQVR2WfX+fj7zgZnnmee5oISLe7kupmu1NyHrTym9SggJauw4\npTSsvQERQv4E0Fh1yXcAXASQCy4J3AjAilI6r5FrLACwAADs7e1909LYjl3myUMpxdWsIqQVVUBu\nrAtPU9mDSVlJMnB+ClBwDXBfAXh/CPBF3RdwU0pLga+/Bv79b+D2bW6V/apVXDcAcc+s4aVWA9eu\n3U/Qzp3jOgoA3GxswwRNLu/Uyh+dhiVkDNO12puQnaKUjiCEfEQpXdOZATaFEOII4AilVNnceWyE\njHmSUUpxPasYt4rK4WKkCy+zh5KyB6YwA+p6YfawKUytmhpg3z5uAVd0NDcX+PrrwMKFXCPLHkyj\nAWJj7ydoYWFAVl2pTXPzBxM0L68eNwD4gNpaoKgIMDFhCRkA8Pl8X1dX1/pCqwcPHkxOSkoSb926\n1eLMmTPdswiyTlBQkMuvv/56y9TUtLbh6ytWrLDW09Orff/997Oaei/T8zSXkDVXqdCKEDIIQAgh\n5CdwLZPqUUqvdlyI9xFCrCilGXVPxwOI6Yz7MExvQQhBPwt98AiQXFAGDaXwNte/n5TxJcCALwCL\nYcDF/6krJPstYNspqwoej1DIjYpNnw6cPMntzFyzBvjgAy4pe+01rgt5D8TjcYmWlxe3gZRSICnp\nfoIWHs7VzwW4jQJDhtxP0Hx8uC+9o1DK7RgtKOBmggsKHvz84Y8Pv1Zc3PI9niZisVjzcBmKpKSk\nHjF0GxYW1q0JIdN1mkvI1gFYC8AWwFY8mJBRAMM7KaYthJB+dfdIBbCwk+7DML0GIQR9zfXBIwRJ\nBWXQUMDHQv/BkTL7yYCRDzeFGf5iz57CJITrezRqFHD1Ktea6d//5ppWvvQStwFA2ezAeLcjhJuq\nlMu59p8AkJbGTW2GhXEJ2uHD3Ou6ulx5DW2C5u/Pld8oLm46mWopwaqubj4+XV2uR6iREZcgOjgA\n3t73nxsZcYOTTMvOnDmjs3z5cvuqqiqeRCLRfPvtt7e8vb2rvL293Xfu3Jnq5+dXCQD+/v5uH3/8\n8Z3a2lo0dn5oaKjJkSNHDCsqKni3b98WjxkzpnDbtm13AWD79u3GW7dutaSUkuDg4MIvv/zyHgDY\n2Nh4RUZGxllZWanXrFljuXfvXlMTE5Maa2vrah8fn3IA+OCDD8x37dplxufzqVwurzxy5Mjf3ffd\nYtqruYQsg1I6hhCyjlL6flcFRCmd1VX3YpjehBACpZkMhACJ+dxIma+lwYNJmcwFGBXBlcSI/wTI\nudCzpzABoH9/4IcfgE2buKTs66+B774Dnn8eWL2ay2B6yQItBwfuMXMm9zwzk0vQtCNoa9dyr/P5\n3BRoc5vc+XwucdImT4aGXMeBhgmV9uPDrxkatm5EjiVknKqqKp67u7sCAOzs7KpOnjz5QFsib2/v\nyitXrsQLhUL89ttvstWrV9v+/vvvKRMmTMjfs2ePsZ+fX3paWpowOztbOHTo0PL8/HxeY+cDgEql\n0omOjlZJpVKNi4uLcuXKlVkCgQAbNmywiYqKijMzM1MPGTJE/v333xvOmjWrUBvDuXPndA4cOGB8\n8+ZNVU1NDfr166fQJmShoaGWaWlpN6VSKc3Nze3adhlMh2kuIQsF4AvgHwC6LCFjGKZphBB4msrA\nIwTxeaWglMLXyhC8hgkLXwIM+BywCOKmMI/1Awbu7pm7MBtydORGyNat43ZlfvYZMGwYVyRs9epu\nac30uCwtgcmTuQfA1UK7cIErsSEQPJpYNUyw9PR6TR7aYebNm2cXExPToW0elEplubYpd1Mam7Js\nKD8/nz916lSn1NRUCSGE1tTUEACYPXt2wciRI+X//ve/03fv3m00bty4gubOB4DBgwcXm5iY1AKA\ni4tLZUpKijgnJ0cQGBhYYm1trQaAqVOn5oeFhek1TMjOnDmj9/zzzxfKZDINAIwaNar+mJubW8X4\n8eOdQkJCCmfMmFH/OtO7NJeQ1RBCvgJgQwgJffggpXRZ54XFMExTCCFQmMrAI4AqtxQUhfB7OCkD\nHpzCPDceCDoKWD/XPUG3hYkJN5S0ciU3UrZ1K5fRODtz1Vvnzu383kedxNiY68k+rofnxsyD1qxZ\nYxMUFFRy8uTJlISEBNHw4cPdAMDJyanG0NBQfenSJen+/fuNt23bltbc+QAgEonqx0X5fP4DyVp7\nnTlzJun48eOygwcPGnz88cdWCQkJscKOXLTIdInmErKxAIIBPAcgqmvCYRimtdxNuJGymJwSaGgh\n/K0bScpkLkBwGHByCNekfFQEYNiz12bVk0qBRYuA+fO51kxbtnCV/9evB159lfvcxKS7o2Q6UEsj\nWd2luLiYb2trWw0A27dvN214bOLEifmbNm2yLCkp4QcEBFS0dH5jhgwZUrZ69Wq7jIwMgZmZmXrf\nvn3GS5YsyW54zvDhw0vnzZvn+MEHH2TU1NSQkydPGs6ZMyentrYWKSkponHjxpWMGjWq1M7Ozrio\nqIj/8K5MpudrcmM2pTSXUvoTgBBK6XcPP7owRoZhmiA31kNfc32kl1bi4r0C1GoaWZQklAFBhwGh\nHnD2BaAis+sDfRza1kwXL3Kr5f39uWlNe3tg2TLg1q3ujpB5wq1ZsyZzw4YNth4eHgq1Wv3AsZkz\nZxYcPXrU+MUXX8xvzfmNcXBwqFm/fv29oKAguYeHh6e3t3fZzJkzH5h6HDx4cPn48ePzlUqlZ3Bw\nsGvfvn3LAECtVpOXXnrJSS6XK5RKpeKVV17JZslY79Sa5uJyAF8CsKCUKgkhfcElaR90RYCtweqQ\nMU+7lIIyRGcXw0JXjEBrI/B5jcyC5EcBJ4dyI2QjzgKC3jntB4ArCPbxx8CePVxRrSlTuEKz/ft3\nd2S9CisMyzBd63Gbi+8A8BaAGgCglN4AMK3DomMY5rE5G+nCx8IAWWVV+OtePtSNjZQZ+wKD9gB5\nV4C/ZgNU0/WBdhRPT2DXLm507I03gKNHAV9fIDgY+OOP5rcvMgzD9ECtSch0KKWXH3qt5TFYhmG6\nlJOhDnwtDZBdXo2Iu/lQaxpJuOz+Afj8L3DnFyD63a4PsqPZ2HBry+7c4T7GxQHPPcdVYt2zh+sM\nwDAM0wu0JiHLJYQ4o67BOCFkEoCM5t/CMEx3cDDQwQArQ+RWVOPC3XzUNJaUua8AXBYAqg+BlF1d\nH2RnMDDgpiz//hv45hsuEZs5E3BxAf7zn/tdwhmGYXqo1iRk/wSwHYA7IeQegNcBLOrUqBiGaTc7\nfSn8rQyRX1GDC3fyUVP7UFJGCOD3OWA5Eri8AMg60z2BdgaxGHj5ZeDmTa5MvqMjsHw5twHgnXfu\nN59kGIbpYVpMyCilf1NKgwGYAXCnlA6mlKZ1fmgMw7SXrb4U/tZGKKiswbm7+ah+OCnjCYHBPwP6\nciB8AlCc0D2BdhYeDxg7ltuVefEiMHw48OGHXBn9OXOAvXuBvLzujpJhGKZeiwkZIcSAEPIJgDAA\nZwghWwkhBp0fGsMwj8NGJkGgjRGKq2pw/k7+oyUxRIZA0BEuOTv7AlD5hG50Cwjgun4nJHBFZQ8d\nAqZNA8zMuBIa777L9TVqqTkkwzBMJ2rNlOU3AEoATKl7FAN4QhaeMMyTzUpPAn8rIxRW1UCVW/Lo\nCXpOQNAhoPwuV82/tqrrg+wqrq7Atm1Abi43arZhA9fhe/NmICiIKzIbEgJ88QWQlMR2aj5FCCG+\n8+fPt9U+X7duncWKFSus23u9hQsX2rq4uHguXLjQdsWKFdbr1q2z6JhI2yc1NVU4evToPo0d8/f3\ndwsPD291u6qJEyc67tq1y6jjouuee/RErUnInCml6+umLv+mlL4HoNH/sAzD9DzWMgmcDHSQVFCG\nnPJGEi7TQGDgd0DOeeDS/zz5iQifz42arVsHnD/PTV0eOADMmsXVN1u6FJDLgT59gIULgV9/BQoK\nujtqphOJRCJ67Ngxo4yMjOa617TaDz/8YBofHx+7ffv2ux1xvcfl6OhYc+LEib+7Ow6mea1JyCoI\nIYO1TwghzwCo6LyQGIbpaF7mMugJ+YjMKHx0PRkAOEwF+n4ApO4BYjZ2fYDdycAA+Mc/uIbmKSlA\ncjL3eb9+wI8/ApMmAaamwKBB3KhaRATQiurrTO/B5/Pp7NmzczZt2vTISFZCQoIoMDBQLpfLFQMH\nDpQnJSWJAG4UZ+7cuXY+Pj7utra2XtoRneHDh7uUl5fzlUqlYseOHQ+M8mzdutVUqVR6uLm5KZ57\n7jnnkpISXl5eHt/a2tqrtpYrrl9cXMyztLTsW1VVRRo7v7l7azQaLFy40NbV1dVTLpfX3z8hIUHk\n6urqCQClpaVk7Nixffr06eM5cuRI58rKykZ7adrY2HgtWrTIVi6XK7y8vDxiYmLE2mNhYWF6D98b\nANauXWuhVCo95HK5Yvny5dbae/fp08dz2rRpDi4uLp7PPPOMa2lpKQGAiIgIqbe3t7tcLleMHDnS\nOScnh/9wHEuWLLFxdnb2lMvligULFtg+fPxJ0pqEbDGALwghqYSQVACfg+2yZJheRcDjwc/KEJVq\nDaKziho/yfNtwGkOcHM9kPpD1wbYkzg7A4sXc6NmeXnAuXPcDk2NBti4EXjmGW56c8IEbgqUtW56\nIqxatSp7//79xnl5eQ8kBYsXL7afMWNGXmJiomrq1Kl5ixcvttMey8rKEkZGRsYfPHgwaf369TYA\ncPr06WSxWKyJj49XzZ8//4Gh1RkzZhTExMTEJSQkqNzc3CpCQ0NNTUxMaj08PMqPHTsmA4C9e/ca\nBAUFFYnFYtrY+c3de/fu3YY3b96UxsXFxZ46dSpx3bp1tmlpaQ90Gf/444/NpVKp5u+//4794IMP\n0lUqlW5T3xMDAwN1YmKiauHChdmvvvpqs1/3/v379ZOTkyU3btyIi4uLU12/fl3n+PHjegBw+/Zt\nybJly7KTk5NjDQwManfv3m0EAHPnznXatGnT3cTERJWnp2fFmjVrHpgmzszM5B87dswoKSkpNjEx\nUbVp06YnuuRWi8OzlNLrALwJIfp1z4s7PSqGYTqcsVQEdxM9xOWVwrK4Anb6D7VOIgTw/wooSwUu\nvgzoOgBmz3RLrD2GUAgMHsw93n8fyM8HTp/mugH8/juXtAFcvbNRo7jHs88C+vrdG7eWRsMllRkZ\nQGbmox97moRbdiiraPV6plbRlZbDzanFpuXGxsaayZMn523evNlcKpXWDyNfu3ZN9/jx4ykAsHjx\n4vz33nuvfpQmJCSkkM/nw9fXtzIvL0/Y2HUbioqKkq5bt86mpKSEX1ZWxg8KCioCgMmTJxf8+OOP\nRuPGjSv5+eefjZcsWZLT3PlN3fvcuXOyKVOm5AsEAtjZ2akDAgJKz58/r+Pn51c/q3X+/Hm9ZcuW\nZQNAQEBAhVwuL28q3jlz5uQDwPz58/Pffffd+oSssXufOHFCPzw8XF+hUCgAoLy8nBcfHy/p06dP\ntY2NTdWgQYMqAMDHx6c8NTVVnJeXxy8pKeG/8MILpXX3yJs8efIDy6FMTExqxWKxZurUqY5jx44t\nnDp1ahN/TT4ZWkzICCGbAGyhlBbWPTcC8Aal9Ako880wTxc3Ez1kllXhelYRTKQi6AgfmiHgi4Ah\nvwJ/DATC/wGMugjInLsn2J7I2Jibwpw0iVtrl5jIJWd//AF89x031cnnAwMHch0DRo3iWjrxH5mJ\neTyVlVxNtYyMppOtjAzunMamV/X0ACurjo3pCfDWW29l9e/fXzFt2rRWbTmWSCT1Cy5b6gsNAAsW\nLHD65ZdfkgcOHFgRGhpqEhYWJgOA6dOnF27cuNEmKyuLHxMTozNu3Lji5s5vz73bg8e7P4lGCKm/\nSWP3ppTi9ddfz1i1atUD37uEhASRSCSqP5/P59OKiorWzM5BKBTi+vXrcYcOHdL/5ZdfjL788kvz\nixcvJrb/K+rZWrOAcQyl9G3tE0ppASHkeQAsIWOYXoZHCAZYGeJUai6iMgsx2NYYhDy0hERsAgQd\nBf4IBMJeAEb9BYieug1PLSMEcHPjHq++ClRVAX/9dT9BW7uWexgZcT02tSNo9vaNX49SoLCw8cTq\n4dca22RACGBuDlhacsmWUsl91D7Xfm5pySVk2vf0JK0YyepMFhYWtePGjSv44YcfTKdPn54HAD4+\nPmVff/210T//+c/87du3G/v5+bW77UN5eTnP3t6+pqqqivz000/GVlZWNQBgYGCg6du3b9nChQvt\nR4wYUSQQCJo9vylDhw4t2bFjh9nSpUvzsrOzBZcvX9YLDQ290zABGjx4cOmePXuMQ0JCSq5cuSJJ\nTExsckRy9+7dxps2bcrcuXOnkY+PT1lz9x4zZkzxhg0brBcsWJBvYGCguXXrlrBhIvYwExOTWn19\n/doTJ07ojR49unTnzp0mAwcOfOB7W1RUxCstLeVNnTq1KDg4uNTZ2dmruRh6u9YkZHxCiJhSWgUA\nhBApAHEL72EYpofSEwnQ11wf17KKkFJQDhfjRpaQ6LsCQw8Ap4OBcxOBYSe40TOmaWIxMGwY99i0\nCcjJAU6d4qY2//gD2LePO8/dnStUS+mjiVZVI7tgJZL7yZSHB/debZLV8KO5OSDokE2CT7V33nkn\n87vvvjPTPt+2bdvt2bNnO3766aeWJiYm6t27d6e299pvvvlmur+/v4exsbG6f//+paWlpfVDp1Om\nTCmYN29enyNHjiS05vzGzJo1qzAiIkLPw8PDkxBC33vvvbv29vbqhISE+n+8K1euzJ42bZpTnz59\nPF1cXCoVCkWTiVZBQQFfLpcrRCIR/emnn5rdpTlhwoTi2NhYyYABA9wBQEdHR7Nnz55bAoGgyaRs\n165dtxYvXuywbNkynr29fdWPP/6Y2vB4YWEhf+zYsS5VVVUEADZu3NitCXtnIy0NdRJC1gAYh/u1\nx14GcIhSuqWTY2s1Pz8/GhkZ2d1hMEyvQSnFX/cKkF1ehWcdTGEgbmL5y63vgb9mA87/A/jv6Hkj\nKr0FpYBKdX/0LDwckEobT6wajmZZWXHr0Trp+04IiaKU+nXKxVspOjo61dvb+wmtStx72djYeEVG\nRsZZWVmxLcUdKDo62tTb29uxsWOtWdT/ESEkGkBw3UsbKaW/d2B8DMN0MUII+lsa4FRqLiIzCjHM\n3hR8XiO/9J1mASVJXCkMmSugWNP1wT4JCAE8PbnH8uXdHQ3DMD1Qq8a3KaUnAJzo5FgYhulCEgEf\nPpYGuHivAHF5JVCaNbEz0Os9Lim7/iag5wzYT+raQBmG6XL37t272d0xPG1atdOBYZgnk7WeBI4G\nUiTmlyG3sSr+ADe6E7gLMB0I/DULyL3ctUEyDMM8BVhCxjBPub7m+tAV8hGZUYSaxqr4AwBfAgw9\nCEisgPAQoCyta4NkGIZ5wrWYkBFCXmvNawzD9E7aKv7l6lpEZzdT91liBgw7CtRWAmfHAjWsRjTD\nMExHac0I2ZxGXpv7ODclhEwmhMQSQjSEEL+Hjr1FCEkmhCQQQp57nPswDNM6JlIR3Ez0cLu4AvdK\nmmlVa+DBFY4tjgfOTwU0bAMWwzBMR2gyISOETCeEHAbgRAg51OBxBkD+Y943BsAEAOEP3VMBYBoA\nTwCjAfyXENLBJa4ZhmmMh4keDCVCXMssQoW6tukTLUcAA/4LZJwAopZxJR0Yphfj8/m+7u7uCldX\nV88xY8b00Tbx7ixHjhyRnTx5sr4A4MSJEx0bNunubuHh4Tpz5861a/lMpiM19z9dBICtAOLrPmof\nbwB4rJErSmkcpTShkUMvAviJUlpFKb0FIBmA/+Pci2GY1tFW8a+lFFEZRc23Y3GZD3isApK+BBJC\nuy5IhukE2mbgSUlJsUKhkG7dutWs4XGNRoPa2mb+SGmj06dPy86dO6fXYRfsYEOHDi3/9ttvn+gi\nrD1RkwkZpTSNUnqWUjqQUhrW4HGVUtpZ8xQ2ABr+T3C37jWGYbqATCSA0kwf2eVV+LuwyZ7DnH6b\nAbsJwNXlwN3DXRMgw3SywYMHlyYnJ4sTEhJEjo6OyvHjxzvK5XLPlJQU0fbt243lcrnC1dXVc/Hi\nxfW/m2bMmGGvVCo9XFxcPJcvX26tfd3GxsZr+fLl1gqFwkMulyuuXbsmSUhIEO3evdts27ZtFu7u\n7ooTJ07oAUBYWJiej4+Pu62trZd2tEyj0WDhwoW2rq6unnK5XLFjx476UbR33nnHUi6XK9zc3BRL\nliyxiY2NFSsUCg/t8Zs3b9Y/X7lypZVSqfRwdXX1nD59uoNGw23e8ff3d1u8eLGNl5eXh6Ojo1Ib\ny5EjR2TPPvusCwCsWLHCevLkyY7+/v5utra2Xh988IE5ABQXF/OGDRvm4ubmpnB1dfVsGBvTPq1Z\n1D+BEJJECCkihBQTQkoIIS2u5iWE/EkIiWnk8WJHBE4IWUAIiSSERObk5HTEJRmGAdDHUAcWumLc\nzClGSVUzf3sRHjDwe8DYF4iYDuRf67ogGaYT1NTU4Pfff9f38vKqAIDbt2+Lly5dmpOcnBwrEono\nhg0bbM6ePZuoUqlir127pvv9998bAsAnn3xyLyYmJi4+Pj72woULskuXLkm11zQ1NVWrVKq4efPm\n5WzevNnCzc2tevbs2TmLFi3Kio+PV40ePboUALKysoSRkZHxBw8eTFq/fr0NAOzevdvw5s2b0ri4\nuNhTp04lrlu3zjYtLU34888/6x87dswwKioqPiEhQbV+/fpMT0/PKplMVhsRESEFgO3bt5vOmDEj\nDwBWrVqVHRMTE5eUlBRbUVHB++mnnwy08anVanLz5s24jz766M77779vjUYkJydLwsLCEq9cuRL3\n8ccfW1dVVZH9+/frW1pa1iQkJKiSkpJiJ0yYwHb5PKbWFIbdAmAcpTSuLRemlAa3fNYj7gFoOG9t\nW/daY9f/CsBXANc6qR33YhimEfer+OfgSkYhhjmYgNdU6x6BDhB0CPg9AAgbBzx3CdBhg9pM+5ws\nO2mXV5vXZLPr9jDhm5SP1B3Z7PRbVVUVz93dXQEAAQEBJa+99lpuWlqa0MrKqnrEiBFlAHD+/Hnd\nwMDAEmtrazUATJ06NT8sLExv1qxZhd99953xt99+a6pWq0lOTo4wOjpaEhAQUAEAL730UgEA+Pv7\nlx86dKjJUaSQkJBCPp8PX1/fyry8PCEAnDt3TjZlypR8gUAAOzs7dUBAQOn58+d1zp49K5s5c2au\nTCbTAFxTdACYO3du7o4dO0z9/f3vHDx40OjKlStxAHD8+HHZJ598YllZWckrLCwUKBSKCgBFADB5\n8uQCABg0aFDZqlWrGm1YO2rUqEKpVEqlUqna2Ni45u7du4L+/ftXvPPOO3aLFy+2efHFF4u0iSXT\nfq1ZuJjV1mTsMRwCMI0QIiaEOAFwBcCqUDJMF5MK+PCxMEBhVQ3iclv4OSu1AoKOADVFXFJWw34u\nM72Ldg1ZfHy86rvvvrsjkUgowDXIbum98fHxos8//9wiLCwsMTExUTV8+PCiysrK+t+t2msJBAKq\nVqubbEqqPQ9A8+s3mzFnzpyCM2fOGPz000+GXl5e5ZaWlrXl5eXkjTfecNi/f39KYmKiaubMmblN\nxIfa2tpG4xOLxfUB8fl8qNVq0rdv36qrV6+qvLy8KtauXWuzcuVKq3YFzdRrcoSMEDKh7tNIQshe\nAL8BqC/lTSnd396bEkLGA/gMgBmAo4SQ65TS5yilsYSQnwGoAKgB/JNS2nErKRmGaTUbmRT2+lVI\nyC+FpZ4YJtJG/3jmGPUFntkLhI8DImYAQ/YDPLZBmmmblkayutOQIUPKVq9ebZeRkSEwMzNTGDfT\ndQAAIABJREFU79u3z3jJkiXZBQUFfKlUqjE2Nq69c+eO4OzZswZBQUElzV1LJpPVFhcXt/gPZOjQ\noSU7duwwW7p0aV52drbg8uXLeqGhoXfEYjH917/+Zb1gwYJ8mUymycrK4ltYWNTq6OjQoKCgohUr\nVth//vnnqQBQXl7OAwBLS0t1UVER7/Dhw0bjxo0reNzvR2pqqtDc3Fy9ZMmSfCMjo9qdO3eaPu41\nn3bNTVmOa/B5OYBRDZ5TAO1OyCilBwAcaOLYvwD8q73XZhim43ib6yO3ohqRGYUY7mgKIa+ZQXWb\n5wHfUCByKXB9NdB/a9cFyjCdzMHBoWb9+vX3goKC5JRSEhwcXDhz5sxCAFAqleXOzs5KKyural9f\n3xaHiCdOnFg4adIk5+PHjxv+5z//ud3UebNmzSqMiIjQ8/Dw8CSE0Pfee++uvb292t7evvjq1as6\n/fr18xAKhTQ4OLjo888/vwcAs2fPzj9x4oSRdk2Xqalp7YwZM3I8PDw8zczM1N7e3mUd8f2IioqS\nvvXWW7Y8Hg8CgYD+97//Ze07HhNp79BoT+Ln50cjIyO7OwyGeSLlllcj/E4eHAyk8LU0bPkNka8B\niaHAgC8B10WdHyDTboSQKEqpX8tndp7o6OhUb2/v3O6M4Umybt06i6KiIv6nn36a3t2xMI+Kjo42\n9fb2dmzsWIuL+gkhjRUZKgIQSSk9+JixMQzTw5nqiCA31kVifhmsdCWwlkmaf0P/T4DSFG6kTNcJ\nsGYNNximK4wcOdI5LS1NHBYWltjdsTBt15pF/RIA/QAk1T36gtv9+D+EkP90YmwMw/QQClMZDMQC\nXM0qQmVzVfwBbu3YMz8CBkrgwhSgMKZrgmSYp9zJkydTEhMTVVZWVqynWS/UmoSsL4BnKaWfUUo/\nAxAMwB3AeDy4roxhmCeUtoq/WqNBVGYLVfwBQCgDgg4DAl0gbCxQkdU1gTIMw/RSrUnIjAA0bPGg\nC8C4bvdjVeNvYRjmSaMvFkJppo+ssircKmqhij8A6NpxSVllDhAeAqibaVrOMAzzlGtNQrYFwHVC\nyC5CyLcArgH4X0KILoA/OzM4hmF6FmdDHZjriHAzuwQl1a2YFTH2BQbtAfKuABfnALTFsk4MwzBP\npRYTMkrpTgCDwNUhOwBgMKX0a0ppGaV0VWcHyDBMz0EIga+lIXgEiMwohKY1u7Tt/gH4/C9wex9w\n/a3OD5JhGKYXajIhI4S4133sD8AKXNPvOwAs615jGOYpJBVyVfwLKmsQn9fKqvzuKwDXxUDcFiDx\ni84NkGHa6Pbt24KxY8f2sbOzU3p6enoEBQW53LhxQ9zU+QkJCSJXV1fPx7mnj4+Pe0vnvP/+++Yl\nJSWtmclqtdTUVOHo0aP7AEBERIR07969Bi29h+kazf2HXlH3cWsjj487OS6GYXowW30p7GQSJOSV\nIr+iuuU3EAL4fgbYhACRrwJ3fuv8IBmmFTQaDUJCQlyGDh1acufOnZjY2Ni4zZs330tPTxd21D1q\namoeee3atWvxLb1v+/btFqWlpR2akDk6OtacOHHibwCIjIzUOXr0KEvIeogm/0NTShfUfXy2kcfw\nrguRYZieyNvCABIBD1cyCqHWtGJtmLYchok/EDEdyL3Y+UEyTAuOHDkiEwgEdPXq1Tna1wYOHFgx\nevToUo1Gg4ULF9q6urp6yuVyxY4dOx5pDl5eXk4mTZrkKJfLFR4eHorDhw/LACA0NNRk+PDhLoGB\ngfJBgwa5Pfw+HR0dH+39/f393UaPHt3HycnJMyQkxEmj0eCDDz4wz87OFgYFBckDAgLkALB//379\nfv36uSsUCo8xY8b0KSoq4gGAjY2N1/Lly60VCoWHXC5XXLt2TQIAR48e1XN3d1e4u7srPDw8FAUF\nBTzt6F5lZSX58MMPrQ8fPmzk7u6u2LFjh5GDg4MyPT1dAAC1tbWwt7evf850vhYzb0KIDiHkXULI\nV3XPXQkhYzs/NIZhejIRnwc/K0OU1dTiZnazrfvuE+hwOy+ltlwj8uKkzg2SYVpw48YNqbe3d6Pb\nhnfv3m148+ZNaVxcXOypU6cS161bZ5uWlvbAyNlHH31kTghBYmKi6ocffvh7wYIFjuXl5QQAYmNj\ndQ4ePJhy5cqVhOZiiIuLk37xxRd3kpOTY2/fvi0+efKk3rvvvpttbm5eExYWlnjp0qXEjIwMwaZN\nm6zCw8MTVSpVXP/+/cs3btxoob2GqampWqVSxc2bNy9n8+bNFgCwdetWy9DQ0LT4+HjVxYsX4/X0\n9Or/cpJIJPStt95KHzduXEF8fLxq/vz5BZMmTcr7+uuvjQHg4MGD+h4eHhXW1tasplkXaU3muwtA\nFLiF/QBwD8A+AEc6KyiGYXoHMx0xXI10kVRQBks9Maz0WqjiDwASM+DZ48AfA4GzY4BREYDEvPOD\nZXq8qIxCu+JqtU5HXlNfJCj3tTJsV9Pyc+fOyaZMmZIvEAhgZ2enDggIKD1//ryOn59ffQ2XiIgI\nvVdffTUbAHx8fCqtra2rb968KQGAIUOGFFtYWLRQSRnw8vIqc3Z2rgEAT0/P8pSUFNHD55w9e1Y3\nJSVF4u/v7w4ANTU1pGHfzJdeeqkAAPz9/csPHTpkBACBgYGlK1eutJsyZUr+9OnTC5ydnZsdyl68\neHFuSEiIy7p167K/+eYb07lz57KWVl2oNXPTzpTSLQBqAIBSWg6AdGpUDMP0GgpTGfRFAlzNbEUV\nfy2ZCxB0BKhIB86OBdQd0u+YYdrMy8urIjo6ukOTQC0dHZ1W1XkRi8X125X5fD7UavUjv2MppRg8\neHBxfHy8Kj4+XpWSkhL7888/1zf0lkgkFAAEAgHVvn/Tpk2ZX3/9dVpFRQVvyJAh7tqpzKa4uLjU\nmJqaqg8dOiS7fv267uTJk4ta+7Uyj681I2TVhBApAAoAhBBnsIKwDMPU4fMIBlgb4kxaLq5lFSHQ\n2giEtOJvNtMA4JmfgHPjgfPTgKEHAB5brvI0a+9I1uMYN25cydq1a8nHH39sunLlylwAuHTpkrSg\noIA/dOjQkh07dpgtXbo0Lzs7W3D58mW90NDQOxUVFfWDGc8880zp//3f/xmHhISU3LhxQ5yRkSHq\n27dv5aVLlx47ydPV1a0tKiriWVlZYdiwYWVvvPGGfUxMjFipVFYVFxfzUlNThX379m3y93FsbKzY\n39+/wt/fvyIqKkonJiZG4u/vXz89q6+vX/vwpoF58+blvPLKK04TJ07MEwjYv8eu1JoRsg0ATgCw\nI4TsAXAKwOrODIphmN7FQCyEp6kMGaVVSCtqQ0V+2xDA73Mg/QjXjLw1dc0YpgPxeDwcOnQo5fTp\n0/p2dnZKFxcXzzVr1tjY2NjUzJo1q9DT07PCw8PDc9iwYfL33nvvrr29/QNrqlavXp2t0WiIXC5X\nTJ061Xn79u2pUqm0Q/5HnjNnTu7o0aPlAQEBcmtra/X27dtTp02b1kculyv8/PzctVOjTdmyZYu5\ndkOCUCikkyZNemDEa8yYMSWJiYlS7aJ+AJg+fXpReXk5f8GCBXkd8TUwrUda7EkHgBBiAiAQ3FTl\nRUppj5pX9vPzo5GRkd0dBsM81SilOH83H/kVNRjhaAo9URv+ur7+FqDaDHhvAjxZ8diuQgiJopT6\ndWcM0dHRqd7e3j3qd8rTLDw8XGf58uV2UVFRzW5EYNonOjra1Nvb27GxY63ZZfl/ACYASKGUHulp\nyRjDMD1Du6r4a3n/C3CcAUS/Ddz6vvOCZBimSW+//bbltGnTnDdt2nSvu2N5GrVmynInuEr9nxFC\n/iaE/EoIea2T42IYphfSEfLhbWGA/MoaJOa3soo/ABAeEPANYDEcuDgPyDzVeUEyDNOoTZs2Zaan\np9987rnn2vCPl+korelleQbAvwCsBbADgB+AxZ0cF8MwvZSdTAJbmQRxuaUoqGxFFX8tvggYsh8w\n8ADOTQAKbnRekAzDMD1Ma6YsTwG4AGAqgAQAAyilLfbgYhjm6UQIQT8LA4jrq/i3YepSZAAMOwYI\nZMDZ54GyLt90xzAM0y1aM2V5A0A1ACWAvgCUdWUwGIZhGiXi8+BnaYjS6lrE5BS37c06tlzhWHUJ\nVzi2urBzgmQYhulBWjNluZxSOhTcwv48cJX72U9IhmGaZa4rhouRLv4uLEdmWWXb3mzoBQw5AJQk\nAuHjgVpW+pBhmCdba6YslxJC9gK4BuBFAN8AGNPZgTEM0/t5msogEwkQlVGEKnWripbfZzkcCNgF\nZJ8FLr4M0Da+n2FaSdvoWys0NNRk9uzZ9t0VD/N0ak2hIAmATwBEUUpZk1GGYVqNzyMYYHW/in+A\ntWHrqvhrOc0Ayu8A0W8BuvZAv82dFyzDMEw3as2U5ceU0kssGWMYpj0MJUIoTGVIL61EWnEbqvhr\nKdYArosB1UdA4hcdHyDDNGPixImOu3btMtI+bziatnbtWgulUukhl8sVy5cvt+6eCJknRbc0qiKE\nTAbXkskDgD+lNLLudUcAceB2cwJcV4BF3RAiwzAdSG6si+zyKlzPKoKRWAgDibD1byYE8P0MKL8H\nRC3jFv3bvth5wTJPnaqqKp67u7tC+7yoqIg/cuTIZhtr79+/Xz85OVly48aNOEopgoODXY4fP643\nZswYVsOLaZfu6hwaA26TwPZGjqVQSvt1cTwMw3QiQripy1OpubiUXoBnHUwh5Ldmk3cdHh945kfg\n1LPAhenAiNOAaWDnBcx0i4MHD9plZ2c/dlPuhszNzctffPHFZuuniMViTXx8vEr7PDQ01CQyMlK3\nufecOHFCPzw8XF+hUCgAoLy8nBcfHy9hCRnTXt2SkFFK4wC0bS0JwzC9mkTAh7+1Ic7dycfVrCL4\nW7VxPZlABwg6DPwxCAgbB4yMAPRdOy9ghgEgEAhobW0tAKC2thY1NTUE4Hq3vv766xmrVq1i7QSZ\nDtFdI2TNcSKEXANQDOBdSum57g6IYZiOYaYjhqepDLG5JfhbWg5no2YHIR4lMQeePQH8MZCrUTYq\ngnuNeSK0NJLVHRwcHKqjoqJ0XnnllYIffvjBUK1WEwAYM2ZM8YYNG6wXLFiQb2BgoLl165ZQJBJR\nGxsbtt6aaZc2zBm0DSHkT0JITCOP5hZ/ZACwp5T6AFgB4AdCiH4T119ACIkkhETm5OR0xpfAMEwn\nkBvrwlJXjBvZxcivaENrJS2ZCxB0BKhIB86OBdRlHR8kw9R59dVXcyIiImRubm6KiIgIXalUqgGA\nCRMmFE+ePDl/wIAB7nK5XDF+/HjnwsJCfnfHy/RehNI2tDXp6JsTchbASu2i/rYe1/Lz86ORkc2e\nwjBMD1Jdq8Hp1FxQAMMdTSFuy3oyrbuHgHPjAavngaEHAF5PHPDv2QghUZRSv+6MITo6OtXb25tN\n+zFPhejoaFNvb2/Hxo512ghZexBCzAgh/LrP+wBwBfB390bFMExHE/F58Lc2RKW6FpEZhWjXH4a2\nIYDf50D6ESByKdCNf1wyDMM8rm5JyAgh4wkhdwEMBHCUEPJ73aGhAG4QQq4D+AXAIkppfnfEyDBM\n5zKWitDXXB9ZZVVIzG/ntKPrYkDxJpC8HVCxorEMw/Re3bXL8gCAA428/iuAX7s+IoZhukMfQx3k\nVlQjNrcExlIhzHTEbb+I97+AsttA9NtcjTKnWR0fKMMwTCfrUVOWDMM8XQgh6G9pAD0hH5fTC1Gp\nrm3HRXhA4DeAxbPAxXlA5qmOD5RhGKaTsYSMYZhuJeTxEGBjBLVGg8vphdC0Zy0YXwwM2Q/ouwPn\nJgAFNzo+UIZhmE7EEjKGYbqdgViIfhYGyK2oRlxuSfsuIjIEhh0DBDLg7PNAWY8racUwDNMklpAx\nDNMjOBjowMFAioT8MmSWVrbvIrp2wLPHAXUJVzi2urBjg2SeSL/88ou+o6Oj0t7eXvn2229bal/3\n9/d3c3R0VMrlcoWTk5Pn7Nmz7XNzc/kAkJycLAwICJA7Ozt7uri4eG7cuLG+QnFERITU29vb3d3d\nXaFUKj3OnDmjAwAajQZz5861s7e3V8rlcsX58+cfaBM1dOhQ15SUFGF8fLyob9++7vb29soXXnih\nT2VlJQGAFStWWJubm/d1d3dXODg4KEeNGuUcFRUl0b5/ypQpDm5ubgq5XK4YPXp0n6KiIh4AHD9+\nXE+hUHgIBALfho3SAYDP5/u6u7sr3N3dFcOHD3dpy/ft2rVrEnd3d4WHh4ciNja2yQWgoaGhJrNn\nz7YHgC1btph9/vnnJsCjjdsbc+TIEdnJkyfbWEEasLGx8crIyGjzOvmpU6c6NPyediWWkDEM02P0\nMzeAgViAyIxClNe0Yz0ZABh6AUMOACWJQPh4oLaqY4NknihqtRrLly+3P3bsWGJiYmLsr7/+atzw\nF/Lu3bv/TkxMVMXFxanEYrFmzJgxLgAgFAqxdevWuykpKbFXrlyJ27lzp7n2fatWrbJ955130uPj\n41Vr165NX7NmjR0A7Nu3z+Dvv/+WpKamxnz55ZdpS5Yssdfep7S0lBQUFAicnZ1rVqxYYbt06dKs\n27dvxxgYGKg//fRTU+15ixYtyoqPj1elpaXFTJ48Of+5555zS09PFwDAtm3b7iQkJKgSExNVtra2\n1R999JE5APTp06d6165dqePGjct7+OvX9vGMj49XnT59Orkt37t9+/YZhoSEFMTFxak8PT1b9Q9t\n9erVOUuXLn0kjqacPn1adu7cOb22xPU49u7dm+br69vOvwgfD0vIGIbpMfg8ggBrI2gAXEovaN96\nMgCwHA4E7AKyzwIXXwaopiPDZJ4gZ8+e1XVwcKhSKBTVEomETpgwIf+XX34xfPg8iURCv/zyy7vp\n6emiv/76S+rg4FAzePDgcgAwMjLSODs7V9y+fVsEcJtVioqK+ABQWFjIt7CwqAaAgwcPGs6YMSOP\nx+NhxIgRZcXFxYK0tDQhABw7dkz2zDPPlGg0Gvz111+yl19+uQAA5s2bl3f48OFH4gGA+fPnFwwZ\nMqRo586dxgBgbGysAbiRuIqKCp62V6ybm1t1QEBABY/Xvl/52hE/uVyuGDlypHNOTg5/7969Bl99\n9ZXFt99+axYQECB/+D2ffvqpiaOjo9LLy8sjIiKiPqFasWKF9bp16ywePr/hiFZ4eLiOv7+/W0JC\ngmj37t1m27Zts3B3d1ecOHFCLz09XfDcc885K5VKD6VS6fHHH3/oAkBmZib/mWeecXVxcfGcOnWq\nQ2O1Db/55hujV155xRYANm7caG5ra+sFACqVStS/f393gBsVDQ8P1wEAHR0dn1dffdXGzc1N4e3t\n7X7nzh0BADQVw9GjR/W0o40eHh6KgoKCNn3DWULGMEyPoicSwNfSAAWVNbiZXdz+CznNALw3AWk/\nciUxGKYRd+7cEdnY2NT38LK1ta2+d++eqLFzBQIBPDw8ymNiYh6Y0kpISBCpVCqdoKCgUgAIDQ29\ns27dOltLS8u+a9eutd26des9AMjIyBA6OjrW38vKyqq6QUJm8PzzzxdlZWUJZDJZrVAoBAA4OjpW\nZ2VlNRoPAPj4+JTHx8fXxzNp0iRHMzMz7+TkZMmbb76Z3dLXX11dzVMqlR7e3t7u33//faOJ39y5\nc502bdp0NzExUeXp6VmxZs0a66lTpxbNnj07Z9GiRVmXLl1KbHh+WlqacPPmzdYRERHxV65ciU9M\nTJS2FEdj3NzcqrX3iI+PV40ePbp04cKFditWrMiKiYmJO3DgQMqiRYscAeDNN9+0HjhwYGlycnLs\n+PHjCzMyMh75ngUHB5dcvHhRBgAXLlzQMzQ0VN+6dUt46tQp2cCBAx9ZvFpRUcEbOHBgaUJCgmrg\nwIGln332mRkANBXD1q1bLUNDQ9Pi4+NVFy9ejNfT02vTX4Ks1wjDMD2OjUwKZ8NqpBSWw0RHBFtZ\nu36ec0Vjy24Dqo8AHTtA/s+ODZTpUP+Zd8guLSZHp+UzW89BaVb++jchHbbD4+GRl6KiIt6ECROc\nN2/efEc7QhUaGmr24Ycf3pk7d27h119/bTR37lzHiIiIxMaup3XlyhW9bdu23S0oKGhTP8yH4/nl\nl19S1Wo15s6da//NN98Yvfbaa81ODyYlJd1wcnKqUalUopEjR7r179+/ouH0Y15eHr+kpIT/wgsv\nlALA/Pnz8yZPntynuWuGh4frBgYGllhbW6sBYMKECfmJiYkdsi7rwoUL+klJSfU/EEpLS/lFRUW8\nixcvyvbv358MANOmTStauHDhI2se7O3t1eXl5byCggJeenq6aPLkyXl//PGH7Pz583oTJkx4ZMGp\nUCik06ZNKwIAX1/fsj///FO/uRgCAwNLV65caTdlypT86dOnFzg7O7cpIWMjZAzD9Ehe5vowkghx\nNbMIJdXq9l2EEMDvM8BmHBC1DLh7sGODZHo9Ozu7B0bE7t69+8CIWUNqtRoJCQk6ffv2rQSAqqoq\n8sILLzhPnjw5f86cOfW/0H/99VeT2bNnFwLAvHnzCm7cuKELAFZWVjWpqan198rIyBA5ODjUqFQq\nkZWVVbVEIqEWFhbqkpISfk1NDQAgNTVVpJ3ybMz169d1PDw8HljzJBAIMGPGjPzffvut2QXzAODk\n5FQDAAqFojowMLDk8uXLHZoQtxafz6caDZe/VFRUNJmbUEpx9erVOO26t+zs7BsGBgatTnz8/PxK\nv/jiC1NnZ+fKZ599tvTcuXN6UVFResHBwaUPnysQCKh2mlcgEECtVpPmYti0aVPm119/nVZRUcEb\nMmSI+7Vr19qUhLIRMoZheiQe4daTnU7NweX0AgyzNwWfR9pxIQHwzI/AqeHAhenAiNOAaWDHB8w8\nto4cyWqtoKCgstTUVEl8fLzI0dGxZv/+/cZ79ux5pIdyVVUVee2112ysrKyqAwICKjQaDaZNm+Yg\nl8srN2zYkNXwXDMzs5pjx47Jxo4dW3L48GGZg4NDJQCEhIQU/ve//zWfP39+/pkzZ3RlMlmtg4ND\nzYcffmg2atSoYgDg8XgIDAws2bVrl9GCBQsKvvnmG5OxY8c2ul3422+/NTx37pzBF198cVej0UCl\nUomVSmWVRqPBgQMHDF1dXZtdnJ6Tk8PX09PTSKVSmpGRIYiMjNR7++23MxueY2JiUquvr1974sQJ\nvdGjR5fu3LnTZODAgY8kLw0NHTq0bM2aNXaZmZl8IyMjzYEDB4w8PT0rmnuPra1t9YULF3SmTJlS\n/PPPP9cnkjKZrLa4uLh+1HDw4MHFH374ofnGjRuzAG5926BBgyoCAwNLvv32W5MtW7Zk/Pzzz/oN\n39PQ4MGDSz/88EPrVatWZQwaNKh87ty5MolEojExMWn1LqKmYoiNjRX7+/tX+Pv7V0RFRenExMRI\nfHx8Wr1BgI2QMQzTY+kI+fCzMkRRlRrXs4vafyGBLhB0GJBaA6dGADffB9TlHRco02vV7Za8PXr0\naLmrq6vnP/7xj3w/P7/6X6KzZ8/uI5fLFW5ubp5lZWW848ePJwPAyZMn9X777TeT8+fPy7QLuffu\n3WsAAF9++WXamjVrbN3c3BRr16612bZtWxoATJkypcjBwaHKwcFBuXjxYocvvvgire5aBi+++GL9\n/+Bbt269+9lnn1na29srCwoKBK+99lqu9ph2gbuDg4Nyz549Jr///nuCtbW1mlKK2bNnO2ljzczM\nFG7evDkdAMLCwnQsLCz6Hjt2zGj58uUOLi4ungBw/fp1ibe3t4ebm5siKChI/vrrr2c2tsNw165d\nt9asWWMrl8sVN27ckGqv2xQHB4eaNWvWpAcGBnr4+fm5y+XyFpOSdevWpa9evdpeqVR68Pn8+nnY\niRMnFh49etRQu6j/q6++unP16lVduVyucHZ29vz888/NAGDz5s3pFy5c0HNxcfHcv3+/kZWVVaOj\niiNGjCjNzMwUBQcHlwgEAlhZWVX7+/s3m2A+rKkYtmzZYu7q6uopl8sVQqGQTpo0qU0/tEhjOxF6\nGz8/PxoZGdndYTAM00lic4qRkF8GX0sDOBg8xoxK2R3g6grgzi9c30vvDwHHl7j2S08hQkgUpdSv\nO2OIjo5O9fb2zm35zCdTRUUFGTBggHtMTExcd8fCdL7o6GhTb29vx8aOPZ0/hRiG6VU8TGUwlYpw\nPasIRVU17b+Qrh0wZB8QHA5ILIG/ZgG/BwI5FzouWIZpA6lUSlkyxgAsIWMYphfgEQJ/a0MIeDxc\nSi9AjeYx64qZDwGeuwQM3A1UpAMnBwPnpwCltzomYIZhmDZiCRnDML2CRMCHv7UhSqtrcS2z6JHt\n/m1GeIDTLGBcAuC1Abh3FDjiAVx/E6h5jPpnDMMw7cASMoZheg0zHTE8TWW4W1KJvws7aFG+QBfw\nWg+MSwQcpnI1yw67AslfAZp2tm9iGIZpI5aQMQzTq8iNdWGhK8aN7GLkVzRZnqntdGyAgd8Bz10B\nZG7A5YXACR8g42TH3YNhGKYJLCFjGKZXIYTAz8oQEgEfl9MLUV3bwX0qTfyA4DBg8C+Augw4Mwo4\nOxYoiu/Y+zAMwzTAEjKGYXodMZ+HAGtDVKhrEZlR+PjryR5GCGA/EXhBBfTbAuScA44pgchXgapm\nO9EwvRCfz/d1d3dXuLq6eo4ZM6ZPSUlJm343NmyM7ePj4945UTJPOpaQMQzTKxlLRehrro/Msiok\n5pd1zk34YkCxChiXDLgsAJL+CxxyAeL/DdR24HQp063EYrEmPj5elZSUFCsUCunWrVvNGh7XaDSo\nrW3desJr166xoVSmXVhCxjBMr9XHUAc2MglUuSXILa9q+Q3tJTEDBvwXGHMDMA3gisseU3K9MZ+A\n4trMfYMHDy5NTk4WJyQkiBwdHZXjx493lMvlnikpKaLt27cby+Vyhaurq+fixYttGnu/jo6ODwAc\nOXJE5u/v7zZ69Og+Tk5OniEhIU7aXo3nzp3TGTBggJunp6fH4MGDXdPS0oRd+CUyPRSoroO4AAAN\ngUlEQVRLyBiG6bUIIehvYQBdIbeerFLdybsiDT2BZ08Aw44BRACE/wM4PQIouN6592W6RE1NDX7/\n/Xd9Ly+vCgC4ffu2eOnSpTnJycmxIpGIbtiwwebs2bOJKpUq9tq1a7rff/+9YXPXi4uLk37xxRd3\nkpOTY2/fvi0+efKkXlVVFVm2bJn9wYMHU2JjY+PmzJmTu3LlykaTO+bpwpqLMwzTqwn5PARYG+HM\n7VxcySjEYFtjENKOJuRtYT0GsBzJlca4uQ443h/o8zLg/QEgterce7eXupxLHPMjgfwo7mNPc3Ge\nHQpjHqM3ViMMleUI/KbZpuVVVVU8d3d3BQAEBASUvPbaa7lpaWlCKyur6hEjRpQBwPnz53UDAwNL\nrK2t1QAwderU/LCwML1Zs2Y12vgbALy8vMqcnZ1rAMDT07M8JSVFZGxsrE5KSpIOHz5cDnDToWZm\nZo/RfoJ5UrCEjGGYXs9AIkQ/CwNczSyCKq8Unqayzr8pTwDIl3C9MGM+ABJDgdt7AcVbgPsKQCDt\n/Biaoq4ACqOBvMj7CVixCqB1O1KlVoCRLwBV98XYg2jXkD38uo6OzmNt4RWLxfXz2Xw+H2q1mlBK\niYuLS8X169fZWjPmAd2SkBFC/hfAOADVAFIAvEwpLaw79haA/wFQC2AZpfT37oiRYZjexdFAB3nl\n1UjIK4WJVAhLXUnX3FhkCPT/GHBdBFxfA9x4lxs567cZcJjG7djsTLWVQMGNusSrLvkqigVo3fSt\nxAIw9gPsJnAfjX0BHeu6N3dybG3VwkhWdxoyZEjZ6tWr7TIyMgRmZmbqffv2GS9ZsiS7rdfp27dv\nZX5+vuDPP//UDQ4OLquqqiI3b94U+/n5VXZG3Ezv0V0jZCcBvEUpVRNCPgLwFoA1hBAFgGkAPAFY\nA/iTECKnlLJy2QzDtMjbwgAFlTWIzCjEcAcz6Aj5XXdzmQsw5FcgKwy4uhyIeAlICAX6fwKYDeyY\ne9RWAYU37k855kcBhTEAVXPHxWZc0mUTwtVTM/YFpDadnxQ+BRwcHGrWr19/LygoSE4pJcHBwYUz\nZ85scrqyKRKJhP70008py5Ytsy8pKeHX1taSxYsXZ7GEjCEdXr+nrQEQMh7AJErpjLrRMVBKP6w7\n9juADZTSv5q7hp+fH42M7IHrIRiG6XIl1WqcScuFvkiAofYm4HVHMkI1wK3dQPTbQEUGN1LWbzOg\n69D6a9RWA0U3uaQrTzvydRPQ1C03EpvcH/Ey9uMeOrZtSr4IIVGUUr82fnUdKjo6OtXb2zu3O2Ng\nmK4SHR1t6u3t7djYsZ6whmwegL11n9sAuNjg2N261xiGYVpFJhKgv6UBLqcXIianBH3N9bs+CMID\n+swF7CYBcf/LPe7+xq0tU7wJCB9a46ap4Ua66ke+IoHCm4CmrtaZyIhLuNzf4D6a+AE69mzki2Ge\nIJ2WkBFC/gRg2cihdyilB+vOeQeAGsCedlx/AYAFAGBvb/8YkTIM86SxlUmRZ1iN5IIymEiFsJF1\n0wJ7oR7Q9z3A+RVutCx2E5Cyk2tmzhNziVdeJDcNqamroyY05Ea93JffH/3SdWTJF8M84TotIaOU\nBjd3nBAyF8BYACPo/XnTewDsGpxmW/daY9f/CsBXADdl+bjxMgzzZPEy10d+ZQ2iMotgIBZCT9SN\nEwK6dsCg7wH5q8C1FcCVJdzrQn0u6XJbdj/50uvDki+GeQp11y7L0QBWAwiilJY3OHQIwA+EkE/A\nLep3BXC5G0JkGKaX4xGCAGtDnErNxaX0AgyzNwWf182Jjqk/EHwOyLsEiEwAmTM3vfl002g0GsLj\n8dgf1swTTaPREABNllLprp8EnwOQAThJCLlOCNkGAJTSWAA/gyuOcwLAP9kOS4Zh2ktHKICflSGK\nqtSIzi7q7nA4hACmgYC+K0vGODE5OTkGdb+sGOaJpNFoSE5OjgGAmKbO6ZYRMkqpSzPH/gXgX10Y\nDsMwTzArPQnkxrpIzC+DiVQEB4OOLQTf29RqKLLLq5BR2om9P9tArVa/kpmZ+XVmZqYSrJ0f8+TS\nAIhRq9WvNHVCT9hlyTAM06kUpjLkV9TgelYRDCVCGIifrl7OlepaZJRWIbOsEtllVailgKCHrFPz\n9fXNBhDS3XEwTHdjCRnDME88HiEYYG2I06m5CL+dBwtdMUylIpjqiCATCTq/92UXo5SiuEqNjLJK\nZJRWoaCSq12mI+D/f3v38xtHfYdx/Pl4f3p3sZ3E+eE0BIKUUhBqVbXAAYGCBIJDBQKhiqq3IiGQ\n+ge0f0CVW089ICQqtZVaVEVUSgUSF4TgwAE4IJGEpCGXJiTEGxCyvYvXa396mLVnG3mDF+/sd2b2\n/ZL24Mlk9JlHnvHjmfWs7pitaaFR1fx0OfCUAPpRyABMhOliQQ8d2av/fLWsxXZHl5eiB6NXClOa\nr5U1P13W/gwXtPUNV7Pd0dXlqIS1u9Hbb/dUS7p3vqGFelUzlWzuGzAJKGQAJsZctaT7D++Ru2tl\nbV2LrY6a7Y4WW6u60l/QelfP0l7QVrsbuta7CnZ9ZVVddxVMOlCv6J56Q4caFVWLY/z4KADfG4UM\nwMQxMzXKRTXKRR2bq20VtGa7o2arV9CW44K2r3f1bL5W1kzAgubuWup0dW15VVdXvtWNdnQrslqc\n0pGZaS00KjpQq4R/vAeAoVHIAEy8/oJ252xU0Fpr61rcKmgdfdEraOWCaX66MraCtuGuG+2Ori6v\n6uryt1pZi25FzlaK+tG+hhYaVc1xKxLIPAoZANzEzFQvF1XvFTRJWlnrbpWzZruvoE1Z9B60WvSH\nArMjKEed9Q19uRIVsC9XVrW24ZoyaX+touN76jrUqKpW4lYkkCcUMgDYgXqpqPpsces5Zq21blTO\nWh0ttjv6ovdcr/KUaV+trP1DFrTlvluRzVZHruh26eFGVYcaVR2sl1Wc4jFdQF5RyADge6iVirrj\n/wraupqt1a3bnJsPXi1tXkGbjkraZkFzd33VXtt6NMVSpytJmikXdXxvXQuNqvZWS9yKBCYEhQwA\nRqBWKujobE1Hb1nQllSaMs1VS/pmtavO+oZM0nytrGNzM1qoV1QP+SHoAILhyAeABGxb0NodNVvR\ng1oP1spaaFR1oF5RucCtSGDSUcgAYAxqpYKOlqZ1dGY69CgAUohfywAAAAKjkAEAAARGIQMAAAiM\nQgYAABAYhQwAACAwChkAAEBgFDIAAIDAKGQAAACBmbuHnmHXzGxJ0vkdrDor6ZtdrLfd8p1uc1zm\nJTV3sB5ZxEadRdpykMiiH1nE7nb320IPAUCSu2f+JemjHa736m7W2275TrdJFpOTRdpyIAuy2G0W\nvHjxSv41abcs/73L9bZbvtNtpg1ZxEadRVZzkMiiH1kAGJu83LL8yN1/HnqONCCLGFnEyCJGFjGy\nANIjL1fIXg09QIqQRYwsYmQRI4sYWQApkYsrZAAAAFmWlytkAAAAmUUhAwAACIxCBgAAEFjuC5mZ\nnTCz983sFTM7EXqekMzsnl4Op8zs5dDzhGRmd5nZa2Z2KvQsIUz6/vfjuIhxvgTCSXUhM7M/m9l1\nM/v0puVPmtl5M7toZr/7js24pGVJVUmXk5o1aaPIwt3PuftLkn4p6aEk503SiLK45O4vJDvpeA2T\nSx73v9+QWeTiuBhkyOMlF+dLIItS/VeWZvaIopPDX939vt6ygqQLkh5XdML4UNKvJBUknbxpE7+R\n1HT3DTM7KOmP7v7rcc0/SqPIwt2vm9lTkl6W9Dd3//u45h+lUWXR+3+n3P25cc2epGFycfezvX/P\nzf73GzaLPBwXgwx5vHyWh/MlkEXF0APciru/Z2Z33rT4AUkX3f2SJJnZ65KedveTkn5xi819LamS\nxJzjMKos3P20pNNm9qakTP7gGfH3RW4Mk4uks+OdbryGzSIPx8UgQx4vm98XmT5fAlmU6kI2wA8k\n/bfv68uSHhy0spk9K+kJSXOS/pTsaGM3bBYnJD2r6ET7VqKTjd+wWeyT9AdJPzWz3/eKWx5tm8sE\n7X+/QVmcUH6Pi0EGZZHn8yWQalksZENx9zckvRF6jjRw93clvRt4jFRw9xuSXgo9RyiTvv/9OC5i\nnC+BcFL9pv4Brki6ve/rI71lk4gsYmSxPXKJkUWMLICUyWIh+1DScTM7ZmZlSc9LOh14plDIIkYW\n2yOXGFnEyAJImVQXMjP7h6QPJN1tZpfN7AV370r6raS3JZ2T9E93PxNyznEgixhZbI9cYmQRIwsg\nG1L92AsAAIBJkOorZAAAAJOAQgYAABAYhQwAACAwChkAAEBgFDIAAIDAKGQAAACBUciAhJjZITN7\n3cw+N7OPzewtM/th6LkAAOmT+8+yBEIwM5P0L0l/cffne8t+IumgpAshZwMApA+FDEjGo5LW3P2V\nzQXu/knAeQAAKcYtSyAZ90n6OPQQAIBsoJABAAAERiEDknFG0s9CDwEAyAYKGZCMdyRVzOzFzQVm\n9mMzezjgTACAlKKQAQlwd5f0jKTHeo+9OCPppKRrYScDAKSRRT83AAAAEApXyAAAAAKjkAEAAARG\nIQMAAAiMQgYAABAYhQwAACAwChkAAEBgFDIAAIDAKGQAAACB/Q90FkEd3A6p+QAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116b2e490>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.subplot(111)\n",
    "colors = ['blue', 'green', 'red', 'cyan', 'magenta', 'yellow', 'black', 'pink', 'lightgreen', 'lightblue',\n",
    "         'gray', 'indigo', 'orange']\n",
    "weights, params = [], []\n",
    "\n",
    "for c in np.arange(-4, 6, dtype=float):\n",
    "    lr = LogisticRegression(penalty='l1', C=10**c, random_state=0)\n",
    "    lr.fit(X_train_std, y_train)\n",
    "    weights.append(lr.coef_[1])\n",
    "    params.append(10**c)\n",
    "    \n",
    "weights = np.array(weights)\n",
    "for column, color in zip(range(weights.shape[1]), colors):\n",
    "    plt.plot(params, weights[:, column], label=df_wine.columns[column+1], color=color)\n",
    "    \n",
    "plt.axhline(0, color='black', linestyle='-', linewidth=3)\n",
    "plt.xlim([10**(-5), 10**5])\n",
    "plt.ylabel('weight coefficient')\n",
    "plt.xlabel('C')\n",
    "plt.xscale('log')\n",
    "plt.legend(loc='upper left')\n",
    "ax.legend(loc='upper center', bbox_to_anchor=(1.38, 1.03), ncol=1, fancybox=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以发现，如果C<0.1,正则项威力很大时，所有特征权重都为0，$C=\\frac{1}{\\lambda}$。\n",
    "\n",
    "\n",
    "\n",
    "### 序列特征选择算法\n",
    "\n",
    "另一种减小模型复杂度和避免过拟合的方法是通过特征选择进行维度降低(dimensionality reduction)，这个方法尤其对非正则模型有用。维度降低有两种做法：特征选择(feature selection)和特征抽取(feature extraction)。\n",
    "\n",
    "特征选择会从原始特征集中选择一个子集合。特征抽取是从原始特征空间抽取信息，从而构建一个新的特征子空间。本节，我们学习特征选择算法。在下一章，我们会学到不同的特征抽取方法来将数据集压缩到一个低维度特征子空间。\n",
    "\n",
    "\n",
    "序列特征选择算法属于贪心搜索算法，用于将原始的d维度特征空间降低到k维度特征子空间，其中k<d。\n",
    "\n",
    "特征选择算法的原理是自动选择一个特征子集，子集中的特征都是和问题最相关的特征，这样能够提高计算效率并且由于溢出了不相干特征和噪音也降低了模型的泛化误差。\n",
    "\n",
    "一个经典的序列特征选择算法是序列后向选择(sequential backward selection, SBS),它能够降低原始特征维度提高计算效率，在某些情况下，如果模型过拟合，使用SBS后甚至能提高模型的预测能力。\n",
    "\n",
    "\n",
    "**Note**贪心算法在每一次选择时都会做出局部最优选择，通常产生一个次优全局解，暴力搜索要考虑所有的可能的情况所以会保证得到全局最优解。但由于穷搜的计算复杂度过高，导致其并不是最佳选择。\n",
    "\n",
    "\n",
    "SBS算法的idea很简单：SBS序列地从原始特征集中移除特征，直到新的特征集数目达到事先确定的值。而在移除特征时，我们需要定义评价函数$J$。一个特征的重要性就用特征移除后的评价函数值表示。我们每一次都把那些评价函数值最大的特征移除，也就是那些对评价函数影响最小的特征去掉。所以，SBS算法有以下4个步骤：\n",
    "\n",
    "\n",
    "* 1 初始化k=d，其中d是原始特征维度。\n",
    "* 2 确定那个评价函数最大的特征 $x^{-}=argmax J(X_{k}-x), x\\in X_{k}$。\n",
    "* 3 从$X_{k}$中移除特征$x^{-}$, k=k-1。\n",
    "* 4 如果k等于事先确定的阈值则终止；否则回到步骤2。\n",
    "\n",
    "\n",
    "\n",
    "不过，sklearn目前并没有实现SBS算法，好在算法简单，我们可以自己实现：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from itertools import combinations\n",
    "\n",
    "from sklearn.base import clone\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class SBS():\n",
    "    def __init__(self, estimator, k_features, scoring=accuracy_score, test_size=0.25, random_state=1):\n",
    "        self.scoring = scoring\n",
    "        self.estimator = clone(estimator)\n",
    "        self.k_features = k_features\n",
    "        self.test_size = test_size\n",
    "        self.random_state = random_state\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=self.test_size, random_state=self.random_state)\n",
    "        dim = X_train.shape[1]\n",
    "        self.indices_ = tuple(range(dim))\n",
    "        self.subsets_ = [self.indices_]\n",
    "        score = self._calc_score(X_train, y_train, X_test, y_test, self.indices_)\n",
    "        self.scores_ = [score]\n",
    "        \n",
    "        while dim > self.k_features:\n",
    "            scores = []\n",
    "            subsets = []\n",
    "            \n",
    "            for p in combinations(self.indices_, r=dim-1):\n",
    "                score = self._calc_score(X_train, y_train, X_test, y_test, p)\n",
    "                scores.append(score)\n",
    "                subsets.append(p)\n",
    "            \n",
    "            best = np.argmax(scores)\n",
    "            self.indices_ = subsets[best]\n",
    "            self.subsets_.append(self.indices_)\n",
    "            dim -= 1\n",
    "            self.scores_.append(scores[best])\n",
    "        self.k_score_ = self.scores_[-1]\n",
    "        \n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        return X[:, self.indices_]\n",
    "    \n",
    "    def _calc_score(self, X_train, y_train, X_test, y_test, indices):\n",
    "        self.estimator.fit(X_train[:, indices], y_train)\n",
    "        y_pred = self.estimator.predict(X_test[:, indices])\n",
    "        score = self.scoring(y_test, y_pred)\n",
    "        \n",
    "        return score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上面的代码中，我们定义了k_features参数来设定想要得到的特征子集数。使用accuracy_score来评估模型在特征子集的表现。在fit方法的while循环内，通过itertools.combination创建特征子集然后对其评估，\n",
    "\n",
    "现在我们用KNN作为Estimator来运行SBS算法：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sbs = SBS(knn, k_features=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<__main__.SBS instance at 0x116d7a2d8>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sbs.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然SBS的fit方法中有分割数据集的功能，但我们还是为SBS提供了训练集，然后fit方法将其分割为子训练集和子测试集，而这个子测试集被称为验证集(validation dataset)。\n",
    "\n",
    "SBS算法记录了每一步最优特征子集的成绩，我们画出每个最优特征子集在验证集上的分类准确率："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "k_feat = [len(k) for k in sbs.subsets_]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VdW5//HPkzAPCgjGCsigyOAASESUqlAH0FpRqxV6\n61Btqb3grL34q7UWO2jVWiteKyrVeq0UkVpqUbRCCk6VIDMIhkEhIIgYIRqGJM/vj7ODx5CwDyE7\n++Twfb9e+5W9156eBTl5zh7WWubuiIiI7E1W3AGIiEj6U7IQEZFQShYiIhJKyUJEREIpWYiISCgl\nCxERCRVZsjCzCWa2ycwWV7O+h5m9ZWY7zOyWSuuGmtlyMyswszFRxSgiIqmJ8sriSWDoXtZvAa4D\n7ksuNLNs4GHgHKAXMMLMekUUo4iIpCCyZOHus0gkhOrWb3L3OcCuSqv6AwXuvsrddwITgWFRxSki\nIuEaxB1AFdoDa5OW1wEnVbWhmY0ERgI0bdq0X8eOHaOPbj+Ul5eTlZUZj4kypS6ZUg9QXdJVutdl\nxYoVm929Xdh26ZgsUubu44HxALm5uZ6fnx9zRHuXl5fHoEGD4g6jVmRKXTKlHqC6pKt0r4uZfZDK\ndumY7gqB5EuEDkGZiIjEJB2TxRygm5l1MbNGwHBgaswxiYgc0CK7DWVmzwKDgLZmtg74OdAQwN3/\naGaHAfnAQUC5md0A9HL3rWY2GpgOZAMT3H1JVHGKiEi4yJKFu48IWf8RiVtMVa2bBkyLIi4REdl3\n6XgbSkRE0oyShYiIhFKyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSShYiI\nhFKyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSShYiIhIosWZjZBDPbZGaL\nq1lvZvYHMysws4VmdkLSujIzmx9MU6OKUUREUhPllcWTwNC9rD8H6BZMI4FHktaVuHufYDo/uhBF\nRCQVkSULd58FbNnLJsOAP3vC20ArM/taVPGIiEjNxfnMoj2wNml5XVAG0MTM8s3sbTO7oO5DExGR\nZA3iDqAandy90My6AjPMbJG7r6y8kZmNJHELi5ycHPLy8uo4zH1TXFyc9jGmKlPqkin1ANUlXWVK\nXeJMFoVAx6TlDkEZ7l7xc5WZ5QF9gT2ShbuPB8YD5Obm+qBBg6KNeD/l5eWR7jGmKlPqkin1ANUl\nXWVKXeK8DTUVuDx4K2oA8Jm7bzCz1mbWGMDM2gIDgaUxxikicsCL7MrCzJ4FBgFtzWwd8HOgIYC7\n/xGYBpwLFABfAN8Pdu0JPGpm5SSS2d3urmQhIhKjyJKFu48IWe/AqCrK3wSOiyouERHZd2rBLSIi\noZQsREQklJKFiIiEUrIQEZFQShYiIhJKyUJEREIpWYiISCglCxERCaVkISIioZQsREQklJKFiIiE\nUrIQEZFQShYiIhJKyUJEREIpWYiISCglCxERCaVkISIioZQsREQklJKFiIiEimwMbjObAJwHbHL3\nY6tYb8CDwLnAF8CV7v5usO4K4PZg01+6+1NRxSnp54V5hdw7fTnri0o4vFVTbh3SnQv6to/kHIVF\nJbR/e0Yk50g+j+qyb+eIsi51UY9MFFmyAJ4ExgF/rmb9OUC3YDoJeAQ4yczaAD8HcgEH5prZVHf/\nNMJYJU28MK+Q26YsomRXGQCFRSWMmbKQXWXlfKv34bVyjn8sWM/P/r6Y7bvKIztHXZ1Hddn/c9w2\nZRGAEkYIc/foDm7WGXixmiuLR4E8d382WF4ODKqY3P1HVW1XndzcXM/Pz6/N8GtdXl4egwYNijuM\nWhFVXQbePYPCopJaP67I3rRv1ZQ3xnwjkmOn++fezOa6e27YdlFeWYRpD6xNWl4XlFVXvgczGwmM\nBMjJySEvLy+SQGtLcXFx2seYqqjqsrdEcfHRDWvlHJNX7Ir8HHV1HtWlds5RWFQS2WczUz73cSaL\n/ebu44HxkLiySOfsDen/DWNfRFWXnDf/xcatO/Yob9+qKfddVTvf/N6q5uqlNs9RV+dRXWrnHFkG\nZTk9OaNnTq2cJ1mmfO7jfBuqEOiYtNwhKKuuXDLcztJyGmbt+SvZtGE2tw7pXmvnuXVId5o2zI70\nHHV1HtVl/8/RuEEWOS0bc/VT+Yx5fiHFO0pr7XyZJM5kMRW43BIGAJ+5+wZgOnC2mbU2s9bA2UGZ\nZLj7X1nOuqISvj+wM+1bNcVIfKv8zUXH1erDxwv6tuc3Fx1H+1ZNIaJzVD6P6rJv56AOzlFRj3u+\nfTx5PxnMNacfyV/z13Lug7PJX7Ol1s6ZMdw9kgl4FtgA7CLx3OFq4BrgmmC9AQ8DK4FFQG7SvlcB\nBcH0/VTO169fP093M2fOjDuEWlPbdZm1YpN3+p8X/bYpC2v1uGH0f5Ke4qrLO6s/8a/f85p3GfOi\n3/3SMt++q3S/j5nu/y9AvqfwNzayZxbuPiJkvQOjqlk3AZgQRVySfj4p3sFNkxbQ7dAW/OybveIO\nRw5gJ3Zuw0vXn8YvX1zKI3krmfneJn4/vA89Djso7tBipxbcEit355bnFvBZyS4e+m5fmjbKDt9J\nJEItGjfg7m8fz+OX57K5eAfnP/QG42etpKw8umYG9YGShcTqT2+sYebyj/npuT317U3Sypm9cph+\nw2kM7tGOX097jxGPvc3aLV/EHVZslCwkNkvWf8bdL73HmT0P5fKTO8UdjsgeDmnRmD9+rx/3XdKb\npeu3cs6Ds5mUv7bi2eoBRclCYvHFzlKue3YerZo15LcX9ybRVZhI+jEzLu7XgZdvOJVj2x/ETyYv\nZOTTc9lcvGd7oEymZCGxuOvFpaza/DkPXNqHNs0bxR2OSKgOrZvxlx8M4PZv9uTfKz5myAOzeGXJ\nR3GHVWeULKTOTVu0gWffWcuPTjuSgUe1jTsckZRlZRk/OLUr/xj9dXIOasLIp+fyk8kL2La9+q5K\nMoWShdSpwqISxjy/kN4dW3Hz2UfHHY5IjXQ/rCUvjBrIqMFHMnnuOs55cDb/WfVJ3GFFSslC6kxp\nWTk3TpxPucMfhvehYbZ+/aT+atQgi1uH9OC5a04mO8sY/tjb/HraMnaUlsUdWiTqdUeC8qX6MKDL\nuJkFvLNmCw9c2ptOhzSPOxyRWtGvUxumXXcqv5q2jPGzVjFrxcf87jt9WLFxW50MSlVXlCwyQFUD\nBqXbgC5z1mzhD6+9z4V923Nh3w5xhyNSq5o3bsCvLzyOs3rm8JPnF3LeQ7PJMqM0aMiXjp/JfaX7\nABng3unLdyeKCiW7yrh3+vKYIvqqz77YxQ0T59OhdTPGDjsm7nBEIjO4x6FMv+E0GjXI2p0oKqTT\nZ7ImdGWRAdZXM2BQdeV1yd35f39bxMat25n841No2aT2BuQRSUdtmjdiRzBsa2Xp8JmsKV1ZZIDD\ngy6dK2vRuAG7yqr+pa0rk/LX8s9FG7jp7KPp07FVrLGI1JXqPpM5BzWp40hqj5JFBrh1SHcaZH21\nBXS2Gdt2lHLh/77B+xu3xRJXwaZi7py6lIFHHcI1px0ZSwwicahqkCWAbdt38fLiDTFEtP+ULDLA\n4O6Hkm3QtGHW7gFd7v9Ob/74vX6sL9rONx96nSdeX015HfaauaO0jOuenUeThln87jt9yMpSdx5y\n4KhqIKfbzulO13YtuOb/3uWmSfPZWs8a8umZRQZ46q017Chz/jlqIMccfvBX1vXr1JrbpizkrheX\n8tqyjdx7Se/dv8BRuuel5SzdsJXHL8+t15feIjV1Qd/2XNC3/VfG4L7q61156LX3eThvJf9ZtYX7\nLunNyUceEm+gKdKVRT33+Y5SJryxmjN6HLpHogBo17Ixj12eyz3fPo4Fa4sY+sAspry7LtJeM2cu\n38SEN1ZzxcmdOLNXTmTnEalvGmZncdPZ3Zl8zck0apDFiMfe5pcvLmX7rvRvyKdkUc89858PKPpi\nF6O+cVS125gZl554BC9dfxo9vtaSmyYt4L+feZctn++s9Xg2bdvOLZMW0OOwltx2bs9aP75IJuh7\nRGv+ed3XuWxAJx5/fTXfeuh1Fhd+FndYexVpsjCzoWa23MwKzGxMFes7mdlrZrbQzPLMrEPSujIz\nmx9MU6OMs77avquMx2avZuBRh3DCEa1Dtz/ikGZMHHkyY87pwb+WbeTsB2Yx472NtRZPeblz86QF\nFO8o5aERfWlSxQM+EUlo1qgBd11wLE9d1Z/PSnZxwcNvMG7G+5TG/AZjdSJLFmaWDTwMnAP0AkaY\nWeUBlu8D/uzuxwNjgd8krStx9z7BdH5UcdZnk/LX8vG2HYwaXP1VRWXZWcY1px/J30d9nbYtGnHV\nk/ncNmURn+8o3e94nnh9NbPf38wd3+pFt5yW+308kQPB6Ue345UbT2PosYdx3ysr+M6jb7Fm8+dx\nh7WHKK8s+gMF7r7K3XcCE4FhlbbpBcwI5mdWsV6qsausnEf/vYp+nVpzctd9f0DW6/CD+Pvogfzo\n9K5MnPMh5/5hNnM/2FLjeBat+4zfTn+Pocccxnf7H1Hj44gciFo1a8S4757Ag8P7ULCpmHMenM0z\n//kgrUbks6iCMbOLgaHu/oNg+TLgJHcfnbTNX4D/uPuDZnYR8DzQ1t0/MbNSYD5QCtzt7i9UcY6R\nwEiAnJycfhMnToykLrWluLiYFi1a1MqxZq/bxROLd3Jjv8b0brd/L7Ut31LGY4t28EmJ882uDbng\nqIZ7tNuoLLku20udn79Zwq5yGHtKU1o0qj+vydbm/0ncVJf0tK912bK9nCcW7WDJJ+Uc3y6bq45p\nRKsm0X2vHzx48Fx3zw3d0N0jmYCLgceTli8DxlXa5nBgCjAPeBBYB7QK1rUPfnYF1gBH7u18/fr1\n83Q3c+bMWjlOaVm5D7p3pp/74CwvLy+vlWNu277Lf/LcAu/0Py/6Ob+f5e9t2LrX7ZPrcsuk+d55\nzIv+1srNtRJLXaqt/5N0oLqkp5rUpays3J98Y7V3v32a9/nFdP/nwvW1H1gAyPcU/qZHeRuqEOiY\ntNwhKNvN3de7+0Xu3hf4aVBWFPwsDH6uAvKAvhHGWq/8c9EGVm/+nNGDj6q1satbNG7APRcfz2OX\n57Jp23a+9dDrPDZrVWhDvqkL1vPc3HWMHnwUA2pwO0xE9pSVZVxxSmf+ed2pHNGmGf/9zLvcMHEe\nn5XE15AvymQxB+hmZl3MrBEwHPjKW01m1tbMKmK4DZgQlLc2s8YV2wADgaURxlpvlJc7D88o4KhD\nWzDkmMNq/fhn9cph+g2nMah7O341bRkjHnubdZ9+UeW2a7d8wU+nLOKEI1px/Rndaj0WkQPdke1a\nMPnHp3DDmd34x8INDP39LN4o2BxLLJElC3cvBUYD04FlwCR3X2JmY82s4u2mQcByM1sB5AC/Csp7\nAvlmtoDEg++73V3JAvjXso0s37iNUYOPjKwLjUNaNObRy/px78XHs2T9Vob+fjbP5a/9ysO2snLn\n+onzAHhweF8aaNQ7kUg0zM7ihjOPZsqPT6Fpo2z+6/H/8It/LKnzhnyRdvfh7tOAaZXK7kianwxM\nrmK/N4HjooytPnJ3xs0s4Ig2zfjW8YdHei4z45Lcjgzoegg3P7eAWycv5NWlGznt6LY8kreKwqIS\n4AsuP7kTHds0izQWEYHeHVvxz2tP5Z6X3+NPb6xh9vubGdb7cCbOWVsnI2Tq62A9Mvv9zSxc9xk/\nHnRknX2T79imGRN/OICfntuT15Zt5PYXlgSJIuG5/HW8MK9wL0cQkdrStFE2d55/DE9f3Z9NW7dz\n/6srKCwqwflyNL6oPo+hf3HM7FozC28eLJEbN7OAww5qwkUn1O2wjFlZxg9P68ohLRrvsa6+j/4l\nUh+d2q0dzRvveWMoys9jKl9Pc4A5ZjYp6L6j/rxEn0HeWb2Fd1Zv4Uend6Vxg3i60fh4244qy+vz\n6F8i9dVHn22vsjyqz2NosnD324FuwBPAlcD7ZvZrM9NoNnVo3MwCDmneiOEnxtc6urrRv6orF5Ho\n1PXnMaUb30HDjY+CqRRoDUw2s99GEpV8xYK1Rcxa8TFXn9qFpo3i65yvqtG/mjbM5tYh3WOKSOTA\nVdefx9C3oczseuByYDPwOHCru+8K2ke8D/wkkshkt4dnFnBQkwZcNqBTrHFUvGVx7/TlFBaV0D7i\nty9EpHrJn8e6eBsqlVdn2wAXufsHyYXuXm5m50USley2/KNtvLJ0I9ed0Y2WTRrGHU6Vo3+JSDwq\nPo91IZXbUC8Bu7sjNbODzOwkAHdfFlVgkvDwzAKaN8rm+6d0jjsUETmApZIsHgGKk5aLgzKJ2OrN\nn/PiwvV8b0AnWjdvFHc4InIASyVZmCf18+Du5UTc8lsSHskroGF2Flef2iXuUETkAJdKslhlZteZ\nWcNguh5YFXVgB7rCohKmvFvI8BM7cmjLJnGHIyIHuFSSxTXAKSS6F18HnEQw4JBE59F/rwRg5Olq\nziIi8Qu9neTum0h0Ly51ZNO27Uycs5Zvn9CB9mrwJiJpIJV2Fk2Aq4FjgN33Q9z9qgjjOqA9MXs1\npWXl/HiQripEJD2kchvqaeAwYAjwbxIj3m2LMqgD2aef7+Tptz/gW70Pp3Pb5nGHIyICpJYsjnL3\nnwGfu/tTwDdJPLeQCPzpzTV8sbOM/x50VNyhiIjslkqyqBj0tcjMjgUOBg6NLqQD17btu3jyjdUM\nOSaH7oe1jDscEZHdUmkvMT4Yz+J2EmNotwB+FmlUB6in3/6ArdtLGT1Y41mLSHrZ65VF0FngVnf/\n1N1nuXtXdz/U3R9N5eDB+BfLzazAzMZUsb6Tmb1mZgvNLM/MOiStu8LM3g+mK/a5ZvVMyc4ynpi9\nmtOPbsdxHQ6OOxwRka/Ya7IIWmvXqFdZM8sGHgbOAXoBI8ysV6XN7gP+7O7HA2OB3wT7tgF+TuLZ\nSH/g55k+Wt+z73zIJ5/vZPQ39KxCRNJPKs8s/mVmt5hZRzNrUzGlsF9/oMDdV7n7TmAiMKzSNr2A\nGcH8zKT1Q4BX3X2Lu38KvAoMTeGc9dKO0jLGz1pF/y5tOLFzKv+0IiJ1K5VnFpcGP0cllTnQNWS/\n9sDapOWK1t/JFgAXAQ8CFwItzeyQavbdox9eMxtJ0Jo8JyeHvLy8kJDiVVxcXGWMeWt38dHWnXzv\naNK+DhWqq0t9kyn1ANUlXWVKXVJpwR1lL3a3AOPM7EpgFokuRcpS3dndxwPjAXJzcz3dx1eoagyI\n0rJyfnZ/Hr07HMyobw+kvgxxninjWWRKPUB1SVeZUpdUWnBfXlW5u/85ZNdCoGPScoegLPkY60lc\nWWBmLYBvu3uRmRUCgyrtmxcWa300dcF61m4p4Y7zjqk3iUJEDjyp3IY6MWm+CXAG8C4QlizmAN3M\nrAuJJDEc+G7yBmbWFtgSPEi/DZgQrJoO/DrpofbZwfqMUl7u/G/eSnoc1pIzeqjpioikr1RuQ12b\nvGxmrUg8rA7br9TMRpP4w58NTHD3JWY2Fsh396kkrh5+Y2ZO4jbUqGDfLWZ2F4mEAzDW3bfscZJ6\nbvqSjyjYVMxDI/qSlaWrChFJXzUZxOhzIKXnGO4+DZhWqeyOpPnJwORq9p3Al1caGcfdGTezgK5t\nm3PucV+LOxwRkb1K5ZnFP0i8/QSJV217AZOiDOpAkLf8Y5as38pvLz6ebF1ViEiaS+XK4r6k+VLg\nA3dfF1E8BwR356EZ79O+VVMu7LvHG8EiImknlWTxIbDB3bcDmFlTM+vs7msijSyDvbXqE979sIi7\nhh1Dw+xU2kWKiMQrlb9UzwHlSctlQZnU0LgZBbRr2ZhLcjuGbywikgZSSRYNgu46AAjmG0UXUmZ7\n98NPeXPlJ4w8tStNGmbHHY6ISEpSSRYfm9n5FQtmNgzYHF1Ime3hGQW0btaQ7550RNyhiIikLJVn\nFtcAz5jZuGB5HVBlq27Zuw+2lvHae5u4+ayjad64Jm8ti4jEI5VGeSuBAUF3HLh7ceRRZZgX5hVy\n7/TlFBZtx4BDWuounojUL6G3oczs12bWyt2L3b3YzFqb2S/rIrhM8MK8Qm6bsojCohIg0WDlrn8s\n44V5hXvfUUQkjaTyzOIcdy+qWAjGlzg3upAyy73Tl1Oy66sd6ZbsKuPe6ctjikhEZN+lkiyyzaxx\nxYKZNQUa72V7SVJxRVHZ+mrKRUTSUSpPWZ8BXjOzPwEGXAk8FWVQmaB4Rylj/7Gk2vWHt2pah9GI\niOyfVB5w32NmC4AzSdxynw50ijqw+uyd1Vu4adJ81heVcGbPQ3m9YDPbd33ZrrFpw2xuHdI9xghF\nRPZNqu9vbiSRKC4BVgPPRxZRPbajtIzfvbKC8bNX0bF1Myb96GRyO7dJehuqhPatmnLrkO5coD6h\nRKQeqTZZmNnRwIhg2gz8FTB3H1xHsdUryzZs5ca/zue9j7Yxov8R3P7NnrvbUlzQtz0X9G2fMcMr\nisiBZ29XFu8Bs4Hz3L0AwMxurJOo6pGycmf8rFX87tXlHNy0EROuzOUbPXLiDktEpFbtLVlcRGIo\n1Jlm9jKJ0fE08EKSDz/5gpufm8+cNZ9yzrGH8asLj6NNczW4E5HMU22ycPcXgBfMrDkwDLgBONTM\nHgH+5u6v1FGMacfd+euctdz14lKyzPjdd3pzYd/2mCmXikhmCm1n4e6fu/tf3P1bQAdgHvA/qRzc\nzIaa2XIzKzCzMVWsP8LMZprZPDNbaGbnBuWdzazEzOYH0x/3sV6R+XjbDn7453zGTFnE8R1a8fKN\np3HRCR2UKEQko+1Tb3ZB6+3xwbRXZpYNPAycRaLzwTlmNtXdlyZtdjswyd0fMbNeJMbr7hysW+nu\nffYlvqi9vPgj/t/fFlG8o5SfndeL75/SmSwNiSoiB4Aouz7tDxS4+yoAM5tI4nZWcrJw4KBg/mBg\nfYTx1NjW7bv4xdSlPP/uOo5tfxAPfKcP3XJaxh2WiEidMXeP5sBmFwND3f0HwfJlwEnuPjppm68B\nrwCtgebAme4+18w6A0uAFcBW4HZ3n13FOUYCIwFycnL6TZw4sdbrseyTMh5ftIMt253zjmzIsCMb\n0qCGVxPFxcW0aNGiliOMR6bUJVPqAapLukr3ugwePHiuu+eGbRf3oAojgCfd/X4zOxl42syOBTYA\nR7j7J2bWj8SD9mPcfWvyzu6++5ZYbm6u12Ybhu27yrhv+nKeyF9N50Oa89hVvTnhiNb7dcxMameR\nKXXJlHqA6pKuMqUuUSaLQiB5kOkOQVmyq4GhAO7+lpk1Adq6+yZgR1A+18xWAkcD+RHGu9viws+4\nadJ8Vmws5rIBnbjt3B40axR3XhURiU+UfwHnAN3MrAuJJDEc+G6lbT4EzgCeNLOeQBMSw7i2A7a4\ne5mZdQW6AauiCLKiK471RSV8rVUT+nZsxStLN9K6WSOe/P6JDOp+aBSnFRGpVyJLFu5eamajSXQ8\nmA1McPclZjYWyHf3qcDNwGNBy3AHrnR3N7PTgLFmtgsoB65x9y21HWPFwEQV402sL9rO+qKP6NPx\nYP50ZX9aq4GdiAgQ8TMLd59G4nXY5LI7kuaXAgOr2O956qCzwqoGJoJEWwolChGRL6Uy+FHGqm4A\novVF2+s4EhGR9HZAJ4vqBiDSwEQiIl91QCeLW4d0p2nD7K+UaWAiEZE9HdDvg1YMQFTxNtThGphI\nRKRKB3SygC8HJhIRkeod0LehREQkNUoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSS\nhYiIhFKyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQkVabIws6FmttzMCsxsTBXrjzCzmWY2z8wW\nmtm5SetuC/ZbbmZDooxTRET2LrIuys0sG3gYOAtYB8wxs6nBuNsVbgcmufsjZtaLxHjdnYP54cAx\nwOHAv8zsaHffc8BsERGJXJRXFv2BAndf5e47gYnAsErbOHBQMH8wsD6YHwZMdPcd7r4aKAiOJyIi\nMYhy8KP2wNqk5XXASZW2uRN4xcyuBZoDZybt+3alffcYocjMRgIjAXJycsjLy6uNuCNTXFyc9jGm\nKlPqkin1ANUlXWVKXeIeKW8E8KS7329mJwNPm9mxqe7s7uOB8QC5ubk+aNCgaKKsJXl5eaR7jKnK\nlLpkSj1AdUlXmVKXKJNFIdAxablDUJbsamAogLu/ZWZNgLYp7isiInUkymcWc4BuZtbFzBqReGA9\ntdI2HwJnAJhZT6AJ8HGw3XAza2xmXYBuwDsRxioiInsR2ZWFu5ea2WhgOpANTHD3JWY2Fsh396nA\nzcBjZnYjiYfdV7q7A0vMbBKwFCgFRulNKBGR+ET6zMLdp5F4HTa57I6k+aXAwGr2/RXwqyjjExGR\n1KgFt4iIhFKyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSShYiIhFKyEBGR\nUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSShYiIhFKyEBGRUJEmCzMbambLzazA\nzMZUsf4BM5sfTCvMrChpXVnSuqlRxikiInsX2RjcZpYNPAycBawD5pjZ1GDcbQDc/cak7a8F+iYd\nosTd+0QVn4iIpC7KK4v+QIG7r3L3ncBEYNheth8BPBthPCIiUkNRJov2wNqk5XVB2R7MrBPQBZiR\nVNzEzPLN7G0zuyC6MEVEJExkt6H20XBgsruXJZV1cvdCM+sKzDCzRe6+MnknMxsJjATIyckhLy+v\nzgKuieLi4rSPMVWZUpdMqQeoLukqU+oSZbIoBDomLXcIyqoyHBiVXODuhcHPVWaWR+J5xspK24wH\nxgPk5ub6oEGDaiPuyOTl5ZHuMaYqU+qSKfUA1SVdZUpdorwNNQfoZmZdzKwRiYSwx1tNZtYDaA28\nlVTW2swaB/NtgYHA0sr7iohI3YjsysLdS81sNDAdyAYmuPsSMxsL5Lt7ReIYDkx0d0/avSfwqJmV\nk0hodye/RSUiInUr0mcW7j4NmFap7I5Ky3dWsd+bwHFRxiYiIqlTC24REQmlZCEiIqGULEREJJSS\nhYiIhFJ4HxrzAAAIp0lEQVSyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJJSS\nhYiIhFKyEBGRUEoWIiISSslCRERCKVmIiEgoJQsREQmlZCEiIqGULEREJFSkycLMhprZcjMrMLMx\nVax/wMzmB9MKMytKWneFmb0fTFdEGaeIiOxdg6gObGbZwMPAWcA6YI6ZTXX3pRXbuPuNSdtfC/QN\n5tsAPwdyAQfmBvt+GlW8IiJSvSivLPoDBe6+yt13AhOBYXvZfgTwbDA/BHjV3bcECeJVYGiEsYqI\nyF5EdmUBtAfWJi2vA06qakMz6wR0AWbsZd/2Vew3EhgZLBab2fL9jDlqbYHNcQdRSzKlLplSD1Bd\n0lW616VTKhtFmSz2xXBgsruX7ctO7j4eGB9NSLXPzPLdPTfuOGpDptQlU+oBqku6ypS6RHkbqhDo\nmLTcISirynC+vAW1r/uKiEjEokwWc4BuZtbFzBqRSAhTK29kZj2A1sBbScXTgbPNrLWZtQbODspE\nRCQGkd2GcvdSMxtN4o98NjDB3ZeY2Vgg390rEsdwYKK7e9K+W8zsLhIJB2Csu2+JKtY6VG9umaUg\nU+qSKfUA1SVdZURdLOlvtIiISJXUgltEREIpWYiISCglizpgZh3NbKaZLTWzJWZ2fdwx7Q8zyzaz\neWb2Ytyx7A8za2Vmk83sPTNbZmYnxx1TTZnZjcHv1mIze9bMmsQdU6rMbIKZbTKzxUllbczs1aC7\nn1eDF13SWjX1uDf4/VpoZn8zs1Zxxrg/lCzqRilws7v3AgYAo8ysV8wx7Y/rgWVxB1ELHgRedvce\nQG/qaZ3MrD1wHZDr7seSeKFkeLxR7ZMn2bOHhjHAa+7eDXgtWE53T7JnPV4FjnX344EVwG11HVRt\nUbKoA+6+wd3fDea3kfijtEeL9PrAzDoA3wQejzuW/WFmBwOnAU8AuPtOdy/a+15prQHQ1MwaAM2A\n9THHkzJ3nwVUfttxGPBUMP8UcEGdBlUDVdXD3V9x99Jg8W0SbcbqJSWLOmZmnUl0mPifeCOpsd8D\nPwHK4w5kP3UBPgb+FNxSe9zMmscdVE24eyFwH/AhsAH4zN1fiTeq/Zbj7huC+Y+AnDiDqSVXAS/F\nHURNKVnUITNrATwP3ODuW+OOZ1+Z2XnAJnefG3cstaABcALwiLv3BT6nftzq2ENwP38YiQR4ONDc\nzL4Xb1S1J2iDVa/f8Tezn5K4Hf1M3LHUlJJFHTGzhiQSxTPuPiXueGpoIHC+ma0h0YvwN8zs/+IN\nqcbWAevcveIKbzKJ5FEfnQmsdveP3X0XMAU4JeaY9tdGM/saQPBzU8zx1JiZXQmcB/yX1+OGbUoW\ndcDMjMS98WXu/ru446kpd7/N3Tu4e2cSD1BnuHu9/Abr7h8Ba82se1B0BrB0L7uksw+BAWbWLPhd\nO4N6+rA+yVSgYtCzK4C/xxhLjZnZUBK3bc939y/ijmd/KFnUjYHAZSS+iVeMDHhu3EEJ1wLPmNlC\noA/w65jjqZHg6mgy8C6wiMTnut50MWFmz5LoG667ma0zs6uBu4GzzOx9EldOd8cZYyqqqcc4oCXw\navC5/2OsQe4HdfchIiKhdGUhIiKhlCxERCSUkoWIiIRSshARkVBKFiIiEkrJQuoFM3Mzuz9p+RYz\nu7OWjv2kmV1cG8cKOc8lQe+2M6tYd2/Qa+y9NThuH72KLVFTspD6YgdwkZm1jTuQZEHHfam6Gvih\nuw+uYt1I4Hh3v7UGYfQB9ilZWII+/5Iy/bJIfVFKoqHZjZVXVL4yMLPi4OcgM/u3mf3dzFaZ2d1m\n9l9m9o6ZLTKzI5MOc6aZ5ZvZiqAPrIpxO+41sznBeAQ/SjrubDObShWtvs1sRHD8xWZ2T1B2B/B1\n4InKVw/BcVoAc83sUjNrZ2bPB+edY2YDg+36m9lbQceHb5pZdzNrBIwFLg0afV1qZnea2S1Jx19s\nZp2DabmZ/RlYDHQ0s7ODY75rZs8F/ZcR/FstDep9377+Z0kGcndNmtJ+AoqBg4A1wMHALcCdwbon\ngYuTtw1+DgKKgK8BjYFC4BfBuuuB3yft/zKJL0/dSPQb1YTEt/3bg20aA/kkOusbRKLjwS5VxHk4\nie432pHorHAGcEGwLo/EmBNV1i9p/i/A14P5I0h0E0NQ/wbB/JnA88H8lcC4pP3vBG5JWl4MdA6m\ncmBAUN4WmAU0D5b/B7gDOARYzpeNdlvF/f+vKf5pXy6hRWLl7luDb8XXASUp7jbHg66uzWwlUNF1\n9yIg+XbQJHcvB943s1VAD+Bs4Pikq5aDSSSTncA77r66ivOdCOS5+8fBOZ8hMW7GCynGC4lE0CvR\nzRMABwXf+A8GnjKzbiR6YW24D8es8IG7vx3MDwB6AW8E52pEoruKz4DtJK6CXgTq9YiIUjuULKS+\n+T2JPpD+lFRWSnBLNbgP3yhp3Y6k+fKk5XK++vtfud8bBwy41t2nJ68ws0EkriyikkXi2//2Sucd\nB8x09wstMS5KXjX77/73CCQPsZoctwGvuvuIygcws/4kOiS8GBgNfGPfqiCZRs8spF5x9y3AJBIP\niyusAfoF8+dTs2/cl5hZVvAcoyuJ2zDTgR8H3ctjZkdb+ABJ7wCnm1lbM8sGRgD/3sdYXiHRySHB\nefsEsweTuJUGiVtPFbaR6KyuwhqC7tbN7AQSt86q8jYw0MyOCrZtHtSxBXCwu08j8Yyo9z7GLxlI\nyULqo/tJ3G+v8BiJP9ALgJOp2bf+D0n8oX8JuCb4Vv84iQfY75rZYuBRQq7Gg1teY4CZwAJgrrvv\na/fa1wG5wcPlpcA1Qflvgd+Y2bxKccwkcdtqvpldSmLclDZmtoTEVcGKamL9mETSedYSPe++ReL2\nW0vgxaDsdeCmfYxfMpB6nRURkVC6shARkVBKFiIiEkrJQkREQilZiIhIKCULEREJpWQhIiKhlCxE\nRCTU/wfYWlCx7xTKEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116bac350>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(k_feat, sbs.scores_, marker='o')\n",
    "plt.ylim([0.7, 1.1])\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Number of features')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我们可以看到，最开始随着特征数目的减少，分类准确率一直在提高，原因可能是降低了维度诅咒。对于k={5,6,7,8,9,10,11}，分类准确率是100%.\n",
    "\n",
    "我们将最优的5维度特征打印出来看一下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "k5 = list(sbs.subsets_[8])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index([u'Alcohol', u'Malic acid', u'Alcalinity of ash', u'Hue', u'Proline'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(df_wine.columns[1:][k5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们使用整个特征维度来检验KNN模型在测试集的分类准确率：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=2, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.983870967742\n"
     ]
    }
   ],
   "source": [
    "print(\"Training accuracy: {}\".format(knn.score(X_train_std, y_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.944444444444\n"
     ]
    }
   ],
   "source": [
    "print(\"Test accuracy: {}\".format(knn.score(X_test_std, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到使用原始的特征集建模，在训练集上分类准确率98%，测试集上分类准确率94%，可能是模型有一丢丢过拟合。\n",
    "\n",
    "我们再使用最优的5维度特征建模看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=2, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn.fit(X_train_std[:, k5], y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.959677419355\n"
     ]
    }
   ],
   "source": [
    "print(\"Training accuracy: {}\".format(knn.score(X_train_std[:, k5], y_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.962962962963\n"
     ]
    }
   ],
   "source": [
    "print(\"Test accuracy: {}\".format(knn.score(X_test_std[:, k5], y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "使用不到一半的原始特征，虽然在训练集上分类准确率下降了，但是在测试集上的表现却提高了！并且此时训练集和测试集准确率相差不多，我们很好地降低了过拟合。\n",
    "\n",
    "**Note** sklearn中提供了很多特征选择算法。包括基于特征参数的递归后向消除法，基于树方法提供的特征重要性的特征选择法和单变量统计检验。具体的可以看sklearn文档。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 利用随机森林评估特征重要性\n",
    "\n",
    "在前面一节，你学习了如何利用L1正则将不相干特征变为0，使用SBS算法进行特征选择。另一种从数据集中选择相关特征的方法是利用随机森林。\n",
    "\n",
    "随机森林能够度量每个特征的重要性，我们可以依据这个重要性指标进而选择最重要的特征。sklearn中已经实现了用随机森林评估特征重要性，在训练好随机森林模型后，直接调用feature_importances属性就能得到每个特征的重要性。\n",
    "\n",
    "下面用Wine数据集为例，我们训练一个包含10000棵决策树的随机森林来评估13个维度特征的重要性(第三章我们就说过，对于基于树的模型，不必对特征进行标准化或归一化):\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feat_labels = df_wine.columns[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "forest = RandomForestClassifier(n_estimators=10000, random_state=0, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=10000, n_jobs=-1, oob_score=False, random_state=0,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "forest.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "importances = forest.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "indices = np.argsort(importances)[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1) Alcohol                        0.182483\n",
      " 2) Malic acid                     0.158610\n",
      " 3) Ash                            0.150948\n",
      " 4) Alcalinity of ash              0.131987\n",
      " 5) Magnesium                      0.106589\n",
      " 6) Total phenols                  0.078243\n",
      " 7) Flavanoids                     0.060718\n",
      " 8) Nonflavanoid phenols           0.032033\n",
      " 9) Proanthocyanins                0.025400\n",
      "10) Color intensity                0.022351\n",
      "11) Hue                            0.022078\n",
      "12) 0D280/0D315 of diluted wines   0.014645\n",
      "13) Proline                        0.013916\n"
     ]
    }
   ],
   "source": [
    "for f in range(X_train.shape[1]):\n",
    "    print(\"%2d) %-*s %f\" % (f+1, 30, feat_labels[f], importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEYCAYAAAAeWvJ8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecnFXZ//HPN/QWkKJIKInSRKlSBRELCKgEgdAsiDwg\nNvCHqFhRrNgeGxYElaJ0gSihiYggAgkEEvqDAQlFOgiIQMj398c5k8xOJruTzV12Zq7367Wv7Nwz\nc19ndjdzzX3KdWSbEEIIoS6j6m5ACCGE/haJKIQQQq0iEYUQQqhVJKIQQgi1ikQUQgihVpGIQggh\n1CoSUQghhFpFIgojiqR7JD0n6Zmmr9UW8pw7SLqvqDZ2GPM3kr5WZcz5kfRlSafW3Y4Q5icSURiJ\n3mV72aavB+psjKRF64y/MLq57aF/RCIKXUPS1pKulvSkpJsk7dB034GSbpP0tKQZkj6Ujy8DXAis\n1nyF1XrF0nrVlK/MPiNpGvCspEXz886R9IikuyUd1mG7x0pybuNMSU9IOlTSFpKm5dfzk6bHf0DS\n3yT9RNJTkm6X9Nam+1eTNFHS45LuknRw031flnS2pFMl/Rs4FPgcsE9+7TcN9vNq/llI+qSkhyU9\nKOnApvuXkvQ9Sf/M7btK0lJD/Y5CmJ/4tBS6gqQxwAXA+4CLgLcC50ha3/YjwMPAO4EZwPbAhZIm\n275B0i7AqbZXbzpfJ2H3A94BPArMBv4AnJ+Prw78SdIdti/u8GVsBayT2zcxv463AYsBUyWdZfuK\npseeDawM7AH8XtI4248DpwM3A6sB6wOXSvqH7T/n544HJgDvB5bI51jb9nub2jLfn1e+f1VgeWAM\nsCNwtqTzbD8BfBd4LfAG4F+5rbM7+B2F0FZcEYWR6Lz8ifpJSeflY+8FJtmeZHu27UuBKcCuALYv\nsP0PJ1cAlwBvXMh2/Mj2TNvPAVsAq9g+xvYLtmcAvwT2XYDzfdX2f21fAjwLnGb7Ydv3A1cCmzY9\n9mHgB7ZftH0GcAfwDklrANsCn8nnuhE4gZR0Gv5u+7z8c3quXUM6+Hm9CByT408CngHWkzQK+CBw\nuO37bb9k+2rbzzPE7yiE+YkrojAS7W77Ty3H1gImSHpX07HFgMsB8lXP0cC6pA9YSwPTF7IdM1vi\nrybpyaZji5ASSKceavr+uTa3l226fb8HViT+J+kKaDXgcdtPt9y3+Xza3VYHP6/HbM9quv2f3L6V\ngSWBf7Q57aC/oxDmJxJR6BYzgVNsH9x6h6QlgHNIVwXn234xX0k1+t/alZh/lvTm27Bqm8c0P28m\ncLftdYbT+GEYI0lNyWhNUnfeA8CKkpZrSkZrAvc3Pbf19Q643cHPazCPAv8FXg3c1HLffH9HIQwm\nuuZCtzgVeJekt0taRNKSeVB9dWBx0ljII8Cs/Gl/p6bnPgSsJGn5pmM3ArtKWlHSqsAnhoh/HfB0\nnsCwVG7D6yRtUdgrHOjlwGGSFpM0AXgNqdtrJnA18M38M9gIOIj085mfh4CxuVsNhv55zZft2cCv\ngO/nSROLSNomJ7fBfkchzFckotAV8hvweNIMsEdIn74/BYzKVwaHAWcCTwD7k64eGs+9HTgNmJHH\nnVYDTiF9or+HND5yxhDxXyIN7m8C3E26MjiBNKBfhmtJExseBb4O7GX7sXzffsBY0tXRucDRbboy\nm52V/31M0g1D/bw6cCSpG28y8DhwLOn3MN/f0QKcO/QhxcZ4IYwskj4A/I/t7epuSwhViE8qIYQQ\nahWJKIQQQq2iay6EEEKt4ooohBBCrUbcOqKVV17ZY8eOrbsZIYQQFtL111//qO1VhnrciEtEY8eO\nZcqUKXU3I4QQwkKS9M9OHhddcyGEEGoViSiEEEKtIhGFEEKo1YgbI1pYv7/jwVLOu8d6ryzlvCGE\n0O/iiiiEEEKtIhGFEEKoVSSiEEIItYpEFEIIoVaRiEIIIdQqElEIIYRaRSIKIYRQq0hEIYQQahWJ\nKIQQQq0iEYUQQqhVz5X4qVIZ5YSilFAIod90dEUkaWdJd0i6S9JRbe7fXtINkmZJ2qvlvpck3Zi/\nJhbV8BBCCL1hyCsiSYsAxwE7AvcBkyVNtH1r08PuBT4AHNnmFM/Z3qSAtoYQQuhBnXTNbQncZXsG\ngKTTgfHAnERk+5583+wS2hhCCKGHdZKIxgAzm27fB2y1ADGWlDQFmAV8y/Z5rQ+QdAhwCMCaa665\nAKfuD7G1RQihl1Uxa24t25sD+wM/kPTq1gfYPt725rY3X2WVVSpoUgghhJGik0R0P7BG0+3V87GO\n2L4//zsD+Auw6QK0L4QQQo/rJBFNBtaRNE7S4sC+QEez3yS9TNIS+fuVgW1pGlsKIYQQhkxEtmcB\nHwMuBm4DzrR9i6RjJO0GIGkLSfcBE4BfSLolP/01wBRJNwGXk8aIIhGFEEKYo6MFrbYnAZNajn2p\n6fvJpC671uddDWy4kG0MIYTQw6LETwghhFpFIgohhFCrSEQhhBBqFUVPwwBRyDWEULW4IgohhFCr\nSEQhhBBqFYkohBBCrSIRhRBCqFUkohBCCLWKRBRCCKFWkYhCCCHUKhJRCCGEWkUiCiGEUKtIRCGE\nEGoViSiEEEKtOqo1J2ln4IfAIsAJtr/Vcv/2wA+AjYB9bZ/ddN8BwBfyza/ZPqmIhofuVkZNO4i6\ndiF0oyGviCQtAhwH7AJsAOwnaYOWh90LfAD4XctzVwSOBrYCtgSOlvSyhW92CCGEXtFJ19yWwF22\nZ9h+ATgdGN/8ANv32J4GzG557tuBS20/bvsJ4FJg5wLaHUIIoUd0kojGADObbt+Xj3Wio+dKOkTS\nFElTHnnkkQ5PHUIIoReMiMkKto+3vbntzVdZZZW6mxNCCKFCnSSi+4E1mm6vno91YmGeG0IIoQ90\nkogmA+tIGidpcWBfYGKH578Y2EnSy/IkhZ3ysRBCCAHoIBHZngV8jJRAbgPOtH2LpGMk7QYgaQtJ\n9wETgF9IuiU/93Hgq6RkNhk4Jh8LIYQQgA7XEdmeBExqOfalpu8nk7rd2j33V8CvFqKNIYQQetiI\nmKwQQgihf0UiCiGEUKtIRCGEEGoViSiEEEKtIhGFEEKoVUez5kLoZmVU+o4q3yEUJ66IQggh1CoS\nUQghhFpFIgohhFCrSEQhhBBqFYkohBBCrSIRhRBCqFUkohBCCLWKdUQhFKSM9UoQa5ZC74srohBC\nCLXqKBFJ2lnSHZLuknRUm/uXkHRGvv9aSWPz8bGSnpN0Y/76ebHNDyGE0O2G7JqTtAhwHLAjcB8w\nWdJE27c2Pewg4Anba0vaFzgW2Cff9w/bmxTc7hBCCD2ikyuiLYG7bM+w/QJwOjC+5THjgZPy92cD\nb5Wk4poZQgihV3WSiMYAM5tu35ePtX2M7VnAU8BK+b5xkqZKukLSG9sFkHSIpCmSpjzyyCML9AJC\nCCF0t7InKzwIrGl7U+AI4HeSRrc+yPbxtje3vfkqq6xScpNCCCGMJJ0kovuBNZpur56PtX2MpEWB\n5YHHbD9v+zEA29cD/wDWXdhGhxBC6B2dJKLJwDqSxklaHNgXmNjymInAAfn7vYA/27akVfJkByS9\nClgHmFFM00MIIfSCIWfN2Z4l6WPAxcAiwK9s3yLpGGCK7YnAicApku4CHiclK4DtgWMkvQjMBg61\n/XgZLySEEEJ36qiygu1JwKSWY19q+v6/wIQ2zzsHOGch2xhCCKGHRWWFEEIItYpEFEIIoVaRiEII\nIdQqqm+H0IXKqPQdVb5DXeKKKIQQQq0iEYUQQqhVdM2FEAYV3YChbJGIQggjQpUJL5LryBJdcyGE\nEGoViSiEEEKtIhGFEEKoVYwRhRBCSWLcqzNxRRRCCKFWkYhCCCHUKhJRCCGEWnWUiCTtLOkOSXdJ\nOqrN/UtIOiPff62ksU33fTYfv0PS24tregghhF4wZCLKW30fB+wCbADsJ2mDlocdBDxhe23gf4Fj\n83M3IO3W+lpgZ+Cnja3DQwghBOjsimhL4C7bM2y/AJwOjG95zHjgpPz92cBbJSkfP93287bvBu7K\n5wshhBCAzqZvjwFmNt2+D9hqfo+xPUvSU8BK+fg1Lc8d0xpA0iHAIfnmM5Lu6Kj1C29l4NEeitOr\nseI1dUeseE3dEavK17RWJw8aEeuIbB8PHF91XElTbG/eK3F6NVa8pu6IFa+pO2JV+Zo61UnX3P3A\nGk23V8/H2j5G0qLA8sBjHT43hBBCH+skEU0G1pE0TtLipMkHE1seMxE4IH+/F/Bn287H982z6sYB\n6wDXFdP0EEIIvWDIrrk85vMx4GJgEeBXtm+RdAwwxfZE4ETgFEl3AY+TkhX5cWcCtwKzgI/afqmk\n1zIcVXUHVtnt2Iux4jV1R6x4Td0Rq/JhkKEoXbiEEEII9YjKCiGEEGoViSiEEEKtIhGFEEY8SSvV\n3YZQnkhEBZK04mBfdbdvYUnaVtKlku6UNEPS3ZJm1N2u4ZK0h6Tl8vdHSTpT0iYVxR4laXRJ5/62\npNGSFpN0maRHJL23jFgVukbSWZJ2zVVbup6kZSSNyt+vK2k3SYvV3a469MVkBUk/Bub7Qm0fVlCc\nu3McAWsCT+TvVwDutT2uiDgtMVcBDgbG0jQL0vYHS4h1O/D/gOuBObMfbT9WQqx3Al8lrcxelPRz\ntO3C3rwlTbO9kaQ3AN8Cvgd81vbWRcVoifc74FDSz24yMBr4oe3vFBznRtubSHo38E7gCOCvtjcu\nMk6ONQG4yPbTkr4AbAZ8zfYNBccR8Dbgg8AWwJnAb2zfWWScpnjbAevY/nX+P7ZsLlNWZIzrgTcC\nLwP+RvqbeMH2e4qMk2O9AvgGsJrtXXId0G1sn1h0rOHolyuiKaQ3z/l9FcL2ONuvAv4EvMv2yrZX\nIr0ZXFJUnBbnkxYQ/wm4oOmrDE/ZvtD2w7Yfa3yVFOsHpLVpK9kebXu5IpNQ1kim7wR+Yft8YImC\nYzTbwPa/gd2BC4FxwPtKiNP4QPIO4CzbT5UQo+GLOQltR0oUJwI/KzqIk0tt70f64HUAcJ2kKyRt\nU2QsSUcDnwE+mw8tBpxaZIxGKNv/AfYAfmp7AqlAdBl+Q1qCs1q+fSfwiZJiLbARUeKnbLZPar4t\nadl8/JmSQm5t++Cm+BdK+nZJsZa2/ZmSzg2ApM3yt5dL+g7we+D5xv1Ff/rNZgI3u9xL9gclHUeq\nDL95XrBd5oezxXLXy+7AT2y/KKmM1/fHfPX6HPDh/In+vyXEgbnJ/B3A8bYvkPS1ooPkMaL3khL3\nQ8DHSQvmNwHOIiX1orwb2BS4AcD2A40u3IIpJ9H3kHYwgLRWswwr2z5T0mdhzvrQEbOmsy8SUYOk\n1wGnACumm3oEeL/tWwoO9UDupmh8inoP8EDBMRr+KGlX25NKOj+kLqtmzXWqDLylhJifBiZJuoKB\nSe/7BcbYG9gV+LHtJyStBsyz31aBfgHcA9wE/FXSWsC/iw5i+6j8wecp2y9JepZ5K+YX5X5JvwB2\nBI6VtATlJPO/k/7v7m77vqbjUyT9vOBYL9h240OCpGUKPn/DJ0hXXefmxf+vAi4vKdazOZk3XtPW\nQJlXygukL8aIGiRdDXze9uX59g7AN2y/oeA4KwJHA9vnQ38FvmL78QJjPM3c8ahlSG/WL1LCWEod\nJF0CPANMB2Y3jtv+SgHnHvRnk7vPKiFpUduzCjrXHoPdb/v3RcRpibk06Ypyuu3/k/RKYEPbhXZF\nS9rb9pktxybYPqvIOPm8R5LKke0IfJM0LvU72z8uOlaOt3TuoitN7tX4MfA64GZgFWAv29PKjNup\nfktEN7UO2LY7FtqTdDjwa+Bp4Jekgemjin7TybFutv26os+bzz2TuUm8lW2vWXC8Iwa7v6irPEm/\nHjxM8RNYctxFgFcwcLLMvQXHuMH2ZkMdKzDejsBOpL+Ri21fWkKMbUhjasvaXlPSxsCHbH+k6Fg5\n3qLAeqTXdIftF8uIMxx91TUHzJD0RdIlPqQ+58KmH0v6ge1PSPoDbWbp2d6tqFhNMbcFbrT9bJ6i\nuxnwg6LfCLIP2v6h0pbvK5H660+hnIkYkyTtVEaSs73G0I8qVGN8YT3SjK9G0eB3UWARYNsHFnWu\nTkn6OOnq/yHmXrka2Kig8+9C6j4dI+lHTXeNJtWvLEVOPIUnnxY/AN5O/nuwfZOk7Qd/ykLZkrmz\nazeThO2TS4zXsX5LRB8EvkIabAe4Mh8rSiPBfbfAcw7lZ8DG+dPUJ4ETcjveVEKsxhXErsDJuV+7\nrDUdHwaOlFRql6OkXZnbhfoX2xcVeX6Y250o6a/AZrafzre/TAkzHCUtz8Cu4SuAY0qaPXc4sF6J\nsycfIM163Y2BM1yfJi0lKFxTtzfA4qRZc8+W0d1te2bLf6FSJhBIOgV4NXBjUwwDkYiqZvsJoJA1\nQ/M5f+M/yhTgOduzYU7XRVnTgmflgdXxpJlYJ0o6aMhnDc/1eexmHPDZPJNo9hDPGRbbZcxSGkDS\n14Ftgd/lQ5+WtJ3tL5QU8hXAC023X8jHivYr0jjA3vn2+0hdqoOOIQ3TTEoc9LZ9E3CTpN8WNZbW\nQcw5f3v5g9Z4oIy1ZTPzGjbn2ZSHA7eVEAfSBKMNSp6FOmz9Nka0LnAk8y7+LHTWl6RrgLc1pofn\n6eKXFD0pIp/7CuAi0pXdG4GHSV11hXSNtMQaRZouO8P2k3kWzpgyBjzn10Vh+68FxpgGbOq8NUnu\nQ7+hjJ9dPv/nScnh3Hxod+AM298sOM6NtjcZ6lhBsU4kdTleQAmzGyWdaXtvSdMZ2N3duEIu5XfV\nph1TbW9a8DlXBn5IWn8lUhf34WVcXUo6CzjM9oNFn7sIfXVFRFpv8HNS91WZc+iXbF6jZPuZPLuo\nDPsA+wMH2v5XfgMvZbqp7dlK1SPWlbRkGTGafKrp+yVJ/dvXU/xU8dGkChgwdyynFLa/LulC0gcG\nSL+zqSWEei5f2V0Fc8YRnyshDsC9+Wvx/FW0w/O/7yzh3G21zD4cRbqaKHwdlu1HSUs7qrAycKuk\n6xj4gaHwcevh6LdENMt24au+23hW0maNhZ6SXk9JbwQ5+VwO7C/pVOBu0iBo4ST9D+mNYXVSX/PW\npPUdha8jsv2ulthrUPzr+jZwg6TLSJ9IdwC+WHAMJI22/e88rf+e/NW4b8Uip/VnhwIn57EikTar\n/EDBMYBiptMPcf7GJ/hHyd3duWdjfVJ1ijI0/+3NIv2+Cl+HpQrLcwFfLuGchemLrjnNLTh6GKnr\n6lwGfioo9I1A0hbA6aSBVgGrAvs0jSEVEWNdYL/89ShwBnCk7bWKitEm5nTSrK9rnGqZrU9ah1XG\n2ENrbAG32N6g4POOAbbKN6+1fX+R588x/mj7nRpYi7DBTmWhCtdYL1XGuqiqZ4iqwrpsVcnrGq9k\n3tqN59TWqJr0SyJq9wbQUMobQR58XC/fLHzOvqTZpD/ig2zflY/NKOtNLZ9/su0tJN0IbGX7eUm3\n2C68PpYGFqptjE3dY7vQKtKSViUVqG3+RHp1kTGqplTdYE/m/aR9TIExXm/7ekltZ2favqKoWDne\nDbY3y9PFl7L97aLHvVRRceSmeKWM27XEuMr2di0zAWGELXzvi645l1D1ugPrARuQxjfKmLO/B7Av\nqf7bRaQrsLLL498naQXgPOBSSU8A/ywp1pSm72cBp9n+W5EBJH2DtJbsNgaugdm1yDgtMXdj4HTx\nP5YQ5nzSTLbrabryL1Lj6r7ohDMIqfy6bM1/c18hTYEvU+nluWxvl/8tfRbqwuiLK6KGfJXyYZre\nCEhVl4u+WjmaNN6wATAJ2AW4yvZeRcbJsZYh9V/vRxqrOZlUu6qsat+NuG8iVf2+yPYLQz1+JJJ0\nB7Cx7bIKgrbG+xapa/O3+dB+wGTbnys4TmlVKdrE2pY0/tC6XUehV+Z5Es6RwN9sH6tUl+0TRV+l\nNMUrfJZcmxhPU3J5Lg2xD1oJ45PD0m+J6ATS4rRGNe73AS/Z/p+C40wHNgam2t5YaS+QU23vWGSc\nNnFfBkwgjUe9tcxYZalyum6+ktzT9rNFnXOIeNOATVrWl00tegqypONJhVynF3ne+cSqbI+qKqnE\n8kFVqmNYYjj6omuuyRYeWFfuz5JuKiFOY3bPrDxg/DBQelkZpwW7x+evblXldN2nSbPm/sTAySuD\n1oZbSCuQZrFBuqIsw3bAB/Kb0POUu+bmKdtlzV6bo6o1gFWQtL7t2zV3e5UBXOC2KjUNSyywfktE\nL0l6te1/AOTL+zLWE03JYym/JH1SfIY0zbkrSVrCdiljDa0qnq57Uf6qyjeBqXm6vUhdxGVsO7FL\nCeecn6r2qCp9DWDLgP7SkhqzDYvuMjsCOIR5t1eB8rZVqWp8clj6rWvuraRSJzNIf1xrkRYVlrUH\nCJLGAqM9QsqtD0fTjKVTbJexo2i7mJVM11XaDG/NxszDsiltk7BFvnmd7X+VFKf0ra5znHb/d1z0\nlYqk622/vshz9pOqxieHq68SEcyZ2to8rbqST/rdTNLNpP3uv8rAigdAafvcVDFd9x3A94HFbY+T\ntAlwtO13FxWjTcwxzB3YB4otW5RjHE2qBrCe7XWVNvw7y/a2RcapklKB2NLXAFZJ0lWkgrRXkiZh\nPF1irErGJ4err7rmJH0U+G3j6kTSyyQdZPunNTdtpDuUNG12BQauOofUlVB4IqKa6brHkBazXg5g\n+0ZJaxccYw5Jx5JKMt3CwOnihSYiqtvqGpiT0F9LWqpAjlnYmqXsgPxv8wchAyNisH2Y3ke66t8T\n+I5SpfkrbZdSVZxqxieHpa8SEXCw7eMaN5y2hz4YiEQ0CKeaZVdJmmL7xIrCVrGN8otOxVubj5XZ\nRbA76Sql7Kvwqra6Rmmb7qWBN5PGb/aiwD2WGrpl0H1B2L5b0n9JVdhfIP0MX1NSuKrGJ4elr7rm\n8pTgjZxfdL48nVZ0ZQCl/eBv8dx9Z0YDr7F9bZFxqpbHUw5l4D43Py96HVZLzNK2UVba0fRC4POk\nJHEYsIztQ0qKdyEwwU0FcUuKU9lW15Km2d6o6d9lgQttv3HIJy9YnKVJg/xr2j5E0jqkpD5iBtwX\nlKR/kCbl/I7UPXdjo+us4Dgi1YecRQXjk8PRb4noO6T++V/kQx8CZtr+ZMFxppI2QGskvFHAlG5f\nl1DVOqwcq/RtlPOVwpdo2hIa+EqJie8c0vqyyxg4zlH4okxVsNV1jnOt7a2Utj7ZA3iM9CGs0C5O\nSWeQZqC+3/brcmK6usgxw6pJOpw01X4N4HbSB7u/Nmb1Fhxruu0Niz5vUfotEY0iJZ/GYs9LgROc\n96MpME67/WCmjZSBweGSdFPLOqy2xwqKdS2pm2diY4V7lRUDyiDpgHbHbZ/U7ng3kPRF4Mek/1PH\nkbo2f2n7SwXHmWJ78+aKB2X97VUtX0UeSFontbrtosdCkXQSaePMyUWfuwh9NUaUL3t/lr/KNEPS\nYU1xPkKaMt7tqlqHBZS/jXKemHAE8y6S3KnIOE3nrSThKO2ncyzwctIVUWkFLm1/NX97jqQ/kvbi\nKmPH1hckLUUew5P0akqqo1cVSd8jXREtC1xNujq/sqRwWwHvlXQP8CzlLnJeYH2RiDRvuZhmLuFT\n1aHAj4Av5LiXkRawdbtPkRYwDliHVVKsKrZRPpvU/Xcq5W6UCEAe1/gmc4vhAuDiy6x8G3iX7bK2\nnZ4jr/f6FWkM6gnKSw5fJi0+XkPSb0lbvJf1t1eVvwPftv1QBbHeXkGMYeuLrjlJ7fboEalv9rO2\nS6u23GuqWoelCrZRVsX1xPK6kaOB/yVNgz8QGFVCN9bfqlozlK8qDyRNS59CWjB+iUt4Y1Hamn5r\n0t/DNU47nIZBKO2kfCiwNjAdONH2rHpbNa++SETNJG1K2lp7Amk303Ns/6Sgc386L7xsu69JGYPS\nYcHlWYyQinU+yLyLJAvfSC7Hvd7265sHjousGKC5W1y/ibQZ43kMfF1lrPdqxB5Fqg/4M9LV5a+B\nHxa14FTSZW4p5NvuWBgoT/J4kdTltwvwT9uHD/6s6vVL11y73Uxl+80Fh2p0hUwZ9FFhSCp3G+Vb\nGFiRuHl7cJM2yivD8/kN+/8kfQy4nzQ+UJTmxcb/Ic2aayhr4TGSNiJdFe0KnEMqI7Md8GfShoYL\nc+4lSeuUVlaqLt/4nY0GxizMufvEBk0fek6khDVeReiLKyLVsJtpWDjqwW2UlbaQv420wv2rpDfT\n79i+ptaGLYQ8RvQkaaztnOauWkm/90JuI5+nOH8CWI2UuBuJ6N+k2XmF9GZUSRXuEdTa/Vx1d3Sn\n+iUR7U7azXRb0oDn6aRp24Wu1pb0Bwbfani3IuNVTdLvSW84F5ax8K4lVhXbKC9Bms6/Hen3diXp\nza2sca/NXHxV6nZxXkUaX9ua9Lr+TtpEroyip6+yXfqMUEkfL2NBbh00cI+gNYEn8vcrAPcW+b4k\n6SXSLDlyjKVIV8sjaqvwvkhEDSp5N1OlXUvny9Vtq1wKSW8jdcFsTSrL/2vbd5QU62ukBYulbaMs\n6XTSGMqp+dD+pAKr+5YU73LS2M3ZwBm2by4pzjWkNT2n5UP7Ah+3vVUJsb5Bmvn1ZL79MuCTtr9Q\nQqw3MG9X7clFx6mKpF+S3n8m5du7ALvb/lC9LateXyWiZuqB3UzrIml5UjL/PDCTtO/SqUWW+lE1\n2yjfanuDoY4VSdKqwN6kWWajSQnpawXHmGfxdIkLj+fZUruM7h9JpwCvBm5kbletu3kCULtqByO9\nAkJZ+jYRlanC9SKVy1No30sq7/MAcwemN7S9Q41NW2CSTgO+31htLun1wBEueM+j+cTeEPg06YPQ\n4gWf+1hSd8/ppC6gfUj7On0HCh+DmEba+fj5fHspUjmrous33kYaeO+ZNyxJF5O6gxtX5O8Btrc9\notf8lKEvZs3V4NfMXS/yZvJ6kVpbVABJ55LWEJ1CWjDZ2E31DEmFzhRU+22UnyJNPy1qHcSGwDW5\nzx5gHHCbUq1Al/Cp/jWkpLAnqSbbGUChdQ6zvfO/rV08+1L81gm/BS5TKiAL6W+9jAoSN5O6NR8c\n6oFdZD/soh7DAAATCUlEQVTS+8S5zN0OZL9aW1STuCIqQdnrReoiadfWMRuVtI14HufYjLQID1LS\nuJm0j8qHixjXy2Vi5ssFF5+U9HfSVcpZth8o8tx1krQzaeExwKW2Ly4hxuWkqeDXMXBtVFdPAII0\ndm372aEf2bviiqgcZa8XqcvXgNbJA38nJYyiPUCabn8LgKQNSBvZfZq0HmahE1HRiaaDeNtUGa9C\nU0lV2Z2/L8OXSzpvbfLkixNI7w2lVJjvFpGIynE4aRHeYaT1Im9h7g6TXScPsI8BlsqVKZoXFS5d\nUth1G0kIwPatkta3PaOlEGrX6MWxQ0l7k8ae/kL6u/ixpE/ZPrvION0+43Q+/pdUA24igO2bJG0/\n+FN6UySiEnhuqfVn6P7CjJD+s3yAtLnW95uOPw18rqSYt0j6GakrC9LYyq157U9pG/GVrBfHDj9P\nmqzwMMypiPEn0hT1hSbpKtvb5VmUzeMII2odzHC55Arz3SLGiAokaeJg93d7f7akPauqbJBnX32E\nNCMP4G+kLd3/Cyztknc5LUMFteYG7SItYzFt63Tj3CV9Uz9OQV5Qks4mfbD7CWmbhsOBzctaxzaS\nxRVRsbYhras5DbiWuV1YXU3Se22fCoyVdETr/ba/3+ZpC8X2c8D38lerhUpCkp6gfQWMxqfsQUuw\nLISyxw4bP6slgc2Bm0ivaSNS/cMyxqguytOQG4tn92HeccTQ3qGkChhjSH8Ll5A+fPWdSETFWhXY\nkTQFc3/gAuC05rGOLrVM/reyCRclj6esXMA5hqPUscNGEd9cimkz29Pz7ddR0mC/7U9J2pNUPgvg\neNvnlhGrB63XumZN0rakq/++El1zJcljGfuRBnK/0o3FGeukivbuybFWZGCy6+qp1ZJuaV1Q2u5Y\nqFe7ChQjtShp2eKKqGA5Ab2DlITGknZq7YlPiCp3a4ZWS9m+TJJs/xP4cq70XFgikvQOUqJbnbTA\ndAxwJ7B+UTFynKqL4U6TdAIDV+xPKzgGQGMPpFK3JZe0CPAnF79tSy0kbQO8AVilpat7NLBIPa2q\nVySiAkk6GXgdqY/8K2UVtazR+aSSJH+i/Nk9VazF+jqpS+kS25tK2pG5VQmK9N0SzjmYA4EPk7oC\nIa3Y/1lJsUrfltz2S5JmS1re9lNlxanQ4qS/5UWB5ZqO/xvYq5YW1Sy65gqktO9RY4V0z001rWJr\nhqZYrXv3LE+q8lzY3j2SptjeXNJNwCa2XUZxUElr2r63yHOOFKpoW3JJ5wObApcy9/9YV+96LGmt\nfLXf9yIRhY5VsTVDlSRdRtoW5FhSt8jDwLa2ty44zpx+f0nn2N6zyPM3xTnT9t6SptN+q/qN2jxt\nuLEq3ZZcUttJHbbLqGtXiVy2qN3v6S01NKdWkYhCxyramqGytViSliNtEjYKeD/pqutk248WFSPH\nmbNVQrttEwqM80rbD0paq939RX76bipyOp9QxY8bSlocWDffvMMFbjtSh1ztvWFJUjHcWbY/XVOT\nahOJKIwokh5hkLVYRZZ6kfQN258b6lgBcZqviCqZFSXpFcAW+eZ1jcoHJcTZ1vbfhjpWQJwdSFW9\n7yH9TawBHGD7r0XGqZuk62xvWXc7qhaJKAwp13i7fX4r94tcsZ9nSDXWYm1EiWux5jN9towxosZ2\nzc1bNUNJY4dt6r+9ESi8/luOVckU5Dxjcn/nHYElrUv6u+jaivZ52UDDKOD1wI9sr1dTk2oTs+ZC\nJz5JmrbdrsqBSQszC2H7JeAi0or9xlqsv0gqbC2WpA+RVrWvK6k5iS4HXF9EjGa2q56SW2r9t3zO\nqqcgL+ambelt3ylpsRLiVOl60v8fAbOAu4GDam1RTSIRhSHZPjj/W8k6jgrWYp0JXEaq3HBU0/Gn\ny+rCqtioltfxGMUXV616CvKUNmujCt2MsWq2x9XdhpEiuubCkJpmSLVV5AyplrVYp5e9FkvSa0ld\nVwBX9kA5JiR9h9St2Vz/bZrtz5QQq5IpyPnDyUeZWwT3SuCnLmFTxrJV+f+pW0QiCkOqcoZUlWux\nJH2U9OZ2Xj40HjjO9k+LilGXlvpvV5ZV/y2P1RzJvNU2+m4KcqfqmHE40kUiCn1L0jTgDc5bSkha\nlrROqrD1Nr0uLwb+OWm8Y061DduFjLXNb01UU5z4XfWAGCMKCyTXZ3stA4uEHlNfixaKgBeabjfW\nRnW1Kuq/NZllu6zyQQDvLPHctWhsq9JuSxUoZ1uVkS4SUeiYpJ+TtjF4M3ACaVD6ulobNQySFrU9\nCzgFuFZSY7O/d5PWqnS70uu/NfmDpI+QJpM0V1Z4vIiTN48/VbU2qgKNbVWWG/RRfSS65kLHJE2z\nvVHTv8sCF9p+45BPHkFaFphuSdMAuOdu8961qqr/lmPd3eawXcy+Uc1xKlsbFaoXV0RhQTyX//2P\npNVI04JfWWN7hmtO95vt6+jCq7ohTJF0BiXXf8vnrGoKculro6oi6UeD3d/NhVyHKxJRWBB/lLQC\n6ZPpDaRB5BPqbdKwtC7CHKAH+uhHk6o37NR0zEAp04KVdoBt3Un35ILDVLE2qiqNiRzbkn5uZ+Tb\nE4Bba2lRzaJrLgxLXtexpLtwfxhJD5L252k7McH2V6ptUfeSdDSwA+kNdRKwC3CV7UIXtc5nbdT0\nbi4QKukaYLs8XkmuFHFl0dXfu0EkotCxvO7mt7afzLdfBuzXbetuqio8WhdJS5JKxbTObiyjIvZ0\nYGNgqu2N84SCU23vWEKsPRg4ntfVOx9LugPYpjGxI/9/uqYfa81166VtqMfBjSQEYPsJUg26btP1\nU7SHcAppj6C3A1eQtkJ/uqRYz9meDcyS1NjTaY2iTi5pbUnbQhrjsn2E7SOARyS9uqg4NfkWMFXS\nbySdROru/kbNbapFJKKwIBaRNOdNPFfKXrzG9gzXW+tuQMnWtv1F4FmnjePeAWxVUqwpedzwl6Sx\njxuAvxd4/h+Q6te1eirf17Vs/5r0ezmXNH63jbt4o7+FEZMVwoK4CDhD0i/y7Q/lY12lqDUuI1hj\nw7gn80SCf5EWtxbO9kfytz+XdBEw2va0AkO8wvb0NnGnSxpbYJxa2P4XcH7d7ahbJKKwID5DSj4f\nzrcvpTtnzfW64/N4wxeBiaQq2V8sK5ik3YDt880rgCIT0QqD3LdUgXFCjWKyQgg9RtIieV+nKmJ9\ni1Tt4Lf50H7A5KJ2uZV0GvBn279sOf4/wI629ykiTpUkjbPdbiFw34pEFIYUhSe7i6R7yd2opDfx\n0v6T58Kxm+QJC41xw6lF/U3kWXjnkmoCNtbfbE4am3x37trqKpKut/16SZfZ7vXxyo5E11zoRM8V\nnuxx65N+Zx8FfiXpD6S9na4qKd4KQGPcbfkiT2z7IeANkt5M2qcK4ALbfy4yTsVGSfocaYfgeRZW\n98CC6gUWiSgMaX4bn0najtQV89FqWxQGY/s/pF1oz8xjRT8kjd2UsYX3N0lTkC8nTYvfnoG73hbC\n9uXA5UWftyb7Arsz7+62fSu65sICkbQpsD+pHMndwO9t/7jeVoVWkt5Eqj6wM2lL7TNsnzP4sxY4\nhkhrlGYxsCp213WX1UHSLrYvrLsdI0EkojCkvAvnfvnrUdLYw5G216q1YaEtSfcAU0lXRRNtPzv4\nMxYq1nTbG5Z1/l4maXngaAbOODymG8tmLaxIRGFIefvuK4GDbN+Vj80outR/KIak0bbbLQItI9ZJ\nwE96YfuMquV9sG5m7h5Y7wM2tr1Hfa2qRySiMCRJu5P6tbclzcY6HTihwi0AwgKouNbc7cA6wD3A\ns8zdDTZmUg5B0o22NxnqWD+IyQphSLbPA86TtAwwHvgE8HJJPwPOtX1JrQ0MrU4BbifVmjsGeA9Q\n1m6tby/pvP3gOUnbNWYz5pp6zw3xnJ4UV0RhWPJsrAnAPrEWYmSRNNX2pk076Ra+vUC+6joUWBuY\nDpzY2M4gdEbSxsDJzJ3y/gRwQMElkrpCJKIQeoyk62xvKemvwEdIteauK3JML+8A+yJp7HAX4J+2\nDy/q/P0kVy2nqnG9kSi65kLoPe1qzX2p4BgbNGbLSTqR3ttuvTL9nIAa4ooohLDAWjcX7PXNBkO5\nIhGF0CPalYtpVmTpGEkvkWbJQZoptxTwH+bOmhtdVKzQ+6JrLoTeUVm5GNtllAvqS5LGAZsCt9q+\nve721CESUQi9Y2nbn5E0wfZZdTcmtCfpPNu75+/Hk3aa/QvwTUnftP2bGptXi+iaC6FH5O06NgKu\nj/GakasxvT5/fzXwHtt3S1oZuMz2xvW2sHpxRRRC77iItBZlWUnNM7Fi3GZkaf70v2hjkzzbj+Zy\nWn0nrohC6DGSzrc9vu52hPaaJnoIWAJYy/aDkhYHpvRjeaRIRCGEMAJIWgF4je2/192WqkUiCqHH\nSNoDOBZ4OelTd3TNhREtElEIPUbSXcC7bJdV6DSUpF/3d4rJCiH0nociCY1c+Yq17V3AqlW2ZaSI\nRBRC75mSi5KeBzzfOGj79/U1KTQ5A/gtA2fPNSzZ5ljPi0QUQu8ZTSq3s1PTMQORiEaGacB3bd/c\neoekt9XQntrFGFEIIVRI0htJ22bc2+a+zW1PqaFZtYpEFEKPkbQ68GPS1u6Q9gw63PZ99bUqhPkb\nVXcDQgiF+zVpH6LV8tcf8rEwAkhaWtKnJX1K0pKSPiBpoqRvS1q27vbVIa6IQugxkm60vclQx0I9\nJJ0JzCRtnbEecBtpAsNuwKq231dj82oRkxVC6D2PSXovcFq+vR/wWI3tCQOta3tvSQIeBN5m25Ku\nAm6quW21iK65EHrPB4G9gX+R3uj2Ag6stUVhHk7dUZPyv43bfdlFFVdEIfQY2/8kdfOEkWmKpGVt\nP2P7g42Dkl4NPF1ju2oTY0Qh9AhJXxrkbtv+amWNCcMiSe7DN+VIRCH0CEmfbHN4GeAgYCXbfTkj\naySStBKwP7B+PnQbcJrtvhzLi0QUQg+StBxwOCkJnQl8z/bD9bYqAEh6DfBn4GJgKqnG3KbAjsBb\nbN9eY/NqEYkohB4iaUXgCOA9wEnAD20/UW+rQjNJZwNn2j6z5fiewP6296ynZfWJRBRCj5D0HWAP\n4HjgONvP1Nyk0IakO2yvt6D39bJIRCH0CEmzSdW2ZzFwGnBsjDeCSLrB9mYLel8vi+nbIfQI27Eu\nsDu8XNIRbY4LWKXqxowEkYhCCKFavwSWm899J1TZkJEiuuZCCCHUKq6IQgihQpIWJU2r3x0Ykw/f\nD5wPnGj7xbraVpe4IgohhApJOg14kjS9vrFH1OrAAcCKtvepq211iUQUQggVknSn7XUX9L5eFrNs\nQgihWo9LmiBpzvuvpFGS9gH6cvFxJKIQQqjWvqStOR6SdKekO0lbduyR7+s70TUXQggVkrSa7Qfy\n9ysB9Gux04ZIRCGEUCFJk4AVgb8AFwFX2Z5Va6NqFokohBAqJmlJYAdgF2Bb4F5SUrrI9r01Nq0W\nkYhCCKFmksaRktLOwKq2t6y5SZWKRBRCCDWQ9AqaFrTafigfX9z2C/W1rHqRiEIIoUKSNgF+DixP\nqqgAaUHrk8CHbU+tq211iUQUQggVknQj8CHb17Yc3xr4he2N62lZfWIdUQghVGuZ1iQEYPsaYJka\n2lO7KHoaQgjVulDSBcDJwMx8bA3g/aSZc30nuuZCCKFiknYFdmNg9e2JtifV16r6RCIKIYRQqxgj\nCiGECknaQtLlkk6VtIakSyU9KWmypE3rbl8dIhGFEEK1jgO+DVwAXE2aKbcCcBTw0zobVpfomgsh\nhApJmmp70/z9vbbXbHdfP4krohBCqNZ/Je0kaQJgSbsDSHoT8FK9TatHTN8OIYRqHUrqmpsNvB34\nsKTfkGbOHVJju2oTXXMhhBBqFV1zIYRQMUnrS3qrpGVbju9cV5vqFIkohBAqJOkw4Hzg48DNksY3\n3f2NelpVrxgjCiGEah0MvN72M5LGAmdLGmv7h4BqbVlNIhGFEEK1Rtl+BsD2PZJ2ICWjtejTRBRd\ncyGEUK2H8p5EAOSk9E5gZWDD2lpVo5g1F0IIFZK0OjDL9r/a3Let7b/V0KxaRSIKIYQaSNoQWD/f\nvM32zXW2p06RiEIIoUKSlifNmlsDmEYaF9oQuBcYb/vfNTavFpGIQgihQpJ+BLwAfNr27HxsFPAt\nYCnbH6+zfXWIRBRCCBWSdCuwke1ZLccXBabbfk09LatPzJoLIYRqvdCahADysedraE/tYh1RCCFU\na8m8AV7rmiEBS9TQntpF11wIIVRI0uWD3W/7zVW1ZaSIRBRCCKFW0TUXQggVk7Q+MB4Ykw/dD5xv\n+/b6WlWfmKwQQggVkvQZ4HTSmNB1+UvA6ZKOqrNtdYmuuRBCqJCkO4HX2n6x5fjiwC2216mnZfWJ\nK6IQQqjWbGC1Nsdfme/rOzFGFEII1foEcJmk/wNm5mNrAmsDH6utVTWKrrkQQqhYLumzJQMnK0y2\n/VJ9rapPJKIQQgi1ijGiEEIItYpEFEIIoVaRiEIIIdQqElEIIYRa/X8RsKWvQCLjLwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117415790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('Feature Importance')\n",
    "plt.bar(range(X_train.shape[1]),\n",
    "       importances[indices],\n",
    "       color='lightblue',\n",
    "       align='center')\n",
    "plt.xticks(range(X_train.shape[1]),\n",
    "          feat_labels, rotation=90)\n",
    "plt.xlim([-1, X_train.shape[1]])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以得出结论：‘Alcohol’是最能区分类别的特征。有趣地是，重要性排名前三的特征也在SBS的最优5特征子集中。\n",
    "\n",
    "sklearn的随机森林实现，包括一个transform方法能够基于用户给定的阈值进行特征选择，所以如果你要用RandomFroestClassifier作为特征选择器，这就很easy了。举个例子：设置阈值为0.15，会选择出三个维度特征，Alcohol、Malic acid和Ash。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lj/.pyenv/versions/2.7.12/lib/python2.7/site-packages/sklearn/utils/deprecation.py:70: DeprecationWarning: Function transform is deprecated; Support to use estimators as feature selectors will be removed in version 0.19. Use SelectFromModel instead.\n",
      "  warnings.warn(msg, category=DeprecationWarning)\n",
      "/Users/lj/.pyenv/versions/2.7.12/lib/python2.7/site-packages/sklearn/utils/__init__.py:54: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(mask.dtype, np.int):\n"
     ]
    }
   ],
   "source": [
    "X_selected = forest.transform(X_train, threshold=0.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(124, 3)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_selected.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 总结\n",
    "\n",
    "本章最开始我们介绍了如何处理缺失值。在训练模型之前，我们必须保证已经正确处理分类数据。\n",
    "\n",
    "\n",
    "此外，我们简单讨论了L1正则，它可以帮助我们降低模型复杂度来避免过拟合。另一种移除不相关特征的方法是使用序列特征选择算法来选择有意义的特征。\n",
    "\n",
    "\n",
    "在下一章，你将会学到另一类降维的方法：特征抽取。它能够将特征压缩到一个低维度子空间而不是移除某些特征。\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
